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Support  Materials 


Introduction 


The  purpose  of  these  support  materials  is  to  facilitate  the  teaching  and  use  of  software  teclinical  reviews  in 
university  courses  on  software  engineering  subjects.  They  arc  intended  to  supplement  the  SEI  cuniculum  mod¬ 
ule,  The  Software  Technical  Review  Process  (CM-3)^  Although  it  has  been  developed  for  use  in  a  university 
setting,  much  of  the  material  is  appropriate  for  the  training  of  professionals  as  well. 

The  user  of  this  material  should  consider  the  variety  of  ways  in  which  technical  review  processes  may  be 
incorporated  into  software-related  curricula.  For  example: 

•  Instruction  in  review  concepts  and  procedures  gives  students  the  necessary  foundation  for  partici¬ 
pating  in  or  implementing  a  technicjil  review  process. 

•  Software  technical  reviews  may  also  be  used  in  the  teaching  of  general  concepts  of  software  engi¬ 
neering  by  requiring  students  to  inspect  and  discuss  concrete  examples  of  software  development 
artifacts. 

•  Practice  in  conducting  reviews  provides  students  with  an  essential  means  of  gaining  comp)etence  as 
reviewers.  Students  may  be  required  to  conduct  reviews  within  the  context  of  software  devel¬ 
opment  projects. 

•  Group  reviews  give  students  an  opportunity  to  gain  experience  and  knowledge  that  will  help  them 
improve  their  effectiveness  as  team  members.  Group  reviews  may  be  used  to  provide  experience 
with  group  interaction,  or  they  may  be  used  as  a  means  of  studying  group  dynamics. 

The  materials  included  here  are  particularly  designed  to  support  an  inspection  metliodology,  or  work  product 
review,  as  it  is  described  in  Section  6  of  CM-3.  These  materials  are  expected  to  meet  the  critical  needs  for 
support  materials  in  this  area.  Practical  knowledge  of  the  inspection  methodologies  of  [Fagan76]  and 
[Freedman82],  together  with  general  knowledge  of  software  development  artifacts,  can  provide  the  basis  for 
using  methodologies  such  as  walk-through  [YourdonSS],  audit  [IEEE85],  in-proccss  software  analysis 
[Howden82],  and  “work  reading"  [Ledgard86]. 

Although  lecture  notes  arc  inclutled,  the  emphasis  is  on  actual  practice  by  groups  of  students.  The  explanatory 
text  defines  basic  i.s.sues  that  an  instructor  should  address  when  structuring  a  learning  activity  that  includes  a 
software  technical  review  process.  Rationale  for  alternative  actions  is  also  given. 

The  instructor  should  consider  the  concerns  listed  in  the  Instructor’s  Checklist  (p.  22)  in  preparing  for  a  software 
technical  review.  Some  support  materials  may  be  used  as  is,  but  others  will  have  to  be  modified  to  suit  the  needs 
of  a  particular  context.  The  introductory  text  provided  with  each  item  attempts  to  put  the  item  in  an  appropriate 
educational  context  and  points  to  specific  modifications  that  might  be  required. 

Part  I  is  designed  to  support  direct  teaching  of  ftmdamental  concepts.  The  part  includes  notes  for  lectures  or 
student  handouts,  sample  software  defects,  and  questions  that  can  be  used  for  knowledge  assessment.  Also 
included  are  guidelines  for  teaching  concepts  using  practice  reviews. 


’Collofello,  James  S.  The  Software  Technical  Review  Process.  Curriculum  Module  SEI-CM-3-1.3,  Software  Engineering  Institute, 
Carnegie  Mellon  University,  Pittsburgh,  Pa.,  April  1988. 
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Part  n,  Implementing  a  Technical  Software  Review  as  a  Learning  Activity,  includes  a  variety  of  materials  to  be 
used  in  preparing  for  practice  reviews  primarily  designed  to  teach  about  software  development.  These  materials 
are  accompanied  by  sample  software  artifacts  to  be  used  for  practice  exercises. 

Part  HI  contains  materials  that  may  be  used  to  facilitate  technical  reviews  of  in-process  software  artifacts, 
particularly  in  the  context  of  a  group  project  Note  that  the  materials  in  Part  in  may  also  be  useful  in  practice 
activities  such  as  those  in  Part  H. 
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Notes  on  Software  Te^^hnical  Reviewing 

John  A.Cross 

Indiana  University  of  Pennsylvania 


Description:  General  notes  on  the  software  technical  review  process  and  the  software  development  life  cycle. 
These  notes  were  used  as  handouts  in  an  undergraduate  course.  Software  Engineering  Concepts,  Computer 
Science  319,  during  three  different  semesters  at  Indiana  University  of  Pennsylvania. 

Purpose:  To  provide  students  with  general  background  material  not  readily  available  in  textbooks. 

Procedure:  These  mtes  can  be  used  as  a  starting  point  for  classroom  lectures  or  they  can  be  given  to  students  as 
handouts.  (Instructors  should  be  aware  that  the  notes  include  several  specific  references  to  Software  Engineering 
Concepts.)  Additional  sources  of  material  appear  in  the  bibliography  of  CM-3,  which  is  reproduced  beginning 
on  p.  73. 

Commentary:  Although  there  are  many  other  sources  of  information  on  software  development  life  cycle  con¬ 
cepts,  the  following  notes  are  included  because  they  contain  concepts  that  are  fundamental  to  software  technical 
reviews  in  a  concise  form.  Deutsch  and  Willis  [Deutsch88]  provide  additional  discussion  of  concepts  of  soft¬ 
ware  technical  reviews  and  software  quality. 
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Notes  on  the  Software  Technical  Review  Process 


Softwrj-e  reviewing  is  a  general  term  applied  to  techniques  for  the  use  of  human  intellectual  power  to  detect 
flaws  in  software  during  the  process  of  software  development.  Because  software  reviewing  does  not  necessarily 
involve  any  automatic  system  execution,  it  is  frequently  categorized  as  a  “static  testing”  approach  to  software 
quality  assurance.  The  most  notable  feature  of  popiilar  techniques  for  software  reviewing  is  the  use  of 
“egoless”  group  interaction.  The  term  egoless  generally  connotes  that  group  members  focus  on  the  task  of 
understanding  a  piece  of  software  and  noting  ways  to  improve  it.  rather  than  on  interpersonal  factors  or  the 
failure  of  an  individual  to  deal  with  possible  defects. 

An  IEEE  draft  standard  for  software  reviews  and  audits  defines  the  basic  concepts  and  terminology  of  software 
reviewing  [IEEE85].  For  example,  a  walk-tlirough  (see  fYourdonSS])  is  a  dynamic  presentation  of  the  behavior 
of  the  software  element  in  various  scenarios.  An  inspection  (see  [Fagan76])  is  a  detailed  technical  review  by 
peers  of  the  authors  of  the  software.  A  management  review  carries  with  it  a  strong  concern  for  development 
schedule  and  allocation  of  organizational  resources.  Finally,  an  audit  emphasizes  independent  evaluation  of 
software  quality.  The  difference  between  formal  and  informal  reviews  is  that  formal  reviews  involve  product 
evaluation  that  is  meant  for  more  than  the  immediate  use  of  the  software  author. 

The  distinction  of  the  IEEE  standard  between  between  formal  and  informal  technical  review  processes  is  a 
fundamental  first  step.  Ledgaid  ([LedgardST],  pp.  65-70)  discusses  informal  “work-reading”  reviews.  This 
approach  and  the  review  of  a  piece  of  software  by  its  author  arc  both  important  “technical  review  processes.” 
However,  the  following  paragraphs  discuss  formal  group  review  processes. 

Formal  review  reports  serve  these  major  functions: 

1.  They  are  used  by  software  authors  to  remove  defects. 

2.  They  are  used  by  management  to  assess  project  .status. 

.  They  provide  a  historical  record  of  software  development. 

The  emphasis  in  the  use  of  software  reviewing  in  this  course  will  be  on  peer  group  technical  reviews.  The 
diagram  below  relates  the  characteristics  of  technical  reviews  to  walk-through  techniques,  which  may  be  more 
familiar  to  you. 

The  goals  of  a  technical  review  include; 

•  Cost-effective  product  enhancement  by  timely  detection  of  defects. 

•  Personal  growth  and  communication  among  software  development  protessionals. 

•  Fostering  teamwork,  professionalism,  participatory  decision-making,  and  high  morale. 

•  Improvement  in  the  ability  of  reviewers  to  prevent  defects  in  'heir  own  work.  (Defect  prevention  is 
a  major  goal  of  any  quality  assunnee  technique.) 

•  Enhancing  the  effectiveness  of  testing  by  detecting  errors  prior  to  testing. 

Varied  complementary  viewpoints  should  be  a  major  goal  in  selecting  the  members  of  a  software  review  group. 
The  persons  in  a  review  group  must  represent  the  interests  of  everyone  who  may  ever  have  a  need  to  use  the 
particular  piece  of  software  which  is  being  reviewed.  For  example,  a  system  specification  must  address  all  of 
the  concerns  of  people  who  understand  the  system  requirements,  it  must  serve  as  a  basis  for  subsequent  phases 
of  system  development,  and  it  must  be  testable  and  modifiable.  This  means  that  the  review  group  should  contain 
some  expertise  in  the  application  area,  technical  expertise  relative  to  subsequent  development  needs,  and  pos¬ 
sibly  specialized  skill  in  testing  or  technical  writing.  Since  students  are  unlikely  to  have  all  of  this  specialized 
expenisc,  student  reviewers  must  make  a  conscious  effort  to  try  to  analyze  software  as  they  think  these  different 
experts  would  see  things. 

In  'ctice,  the  formal  goals  of  a  software  review  are  to  assess  the  quality  of  a  piece  of  software  and  raise  issues 
w,  might  enhance  or  assure  its  quality.  Software  reviewing  exercises  are  also  used  to  provide  learning 
situations  in  which  participants  develop  their  understanding  of  software  development  by  detailed  analysis  and 
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COMPARISON  OF  A  FORMAL  MEETING  AND  A  LECTURE 


Formal  Meeting 

Lecture 

•  Technical  Inspection  Review 

•  Structured  Walk-through 

•  3-7  participants 

•  1  person  to  large  audience 

•  leader  control  high 

•  presenter  leads,  using  scenarios 

•  producer  participation  passive  [Fagan76] 

•  producer  participation  high 

or  none  [Freeclman82] 

•  high  participant  preparation 

•  low  participant  preparation 

•  low  demand  on  presentation  skill 

•  high  demand  on  presentation  skill 

•  product  must  stand  on  its  own 

•  presenter  explains  product 

•  covers  limited  amount  of  material 

•  much  r'ptcrial  can  be  covered 

•  uncovers  many  issues 

•  superficial  detection  of  flaws 

•  high  demand  for  group  interaction 

•  low  demand  for  group  interaction 

discussion  of  examples  of  software  development  documents.  The  immediate  goal  of  a  software  review  in  either 
context  is  to  raise  issues  about  the  software  which  is  being  reviewed.  An  issue  is  generally  something  which  is 
inadequate  about  the  software  relative  to  its  agreed  upon  goals,  but  it  may  also  be  something  so  remarkably 
important  that  it  requires  special  mention.  Reviewers  should  resist  the  urge  to  suggest  how  to  resolve 
issues — that  is  the  job  of  the  software  author.  Reviews  should  raise  issues,  not  resolve  them. 

Questions  of  style  and  minor  errors  should  not  be  discussed  in  review  meetings.  Style  issues  arc  generally  not 
specific  to  the  software  which  is  being  reviewed,  and  they  can  cause  a  meeting  to  become  bogged  down  in 
discussions  which  should  take  place  in  a  broader  context.  Minor  errors  should  simply  be  noted  in  the  reviewed 
document. 

Concerns  about  “correct”  language  use  and  document  form  are  important,  and  poorly  written  communication 
cannot  be  accepted.  Student  reviewers  tend  to  be  especially  sensitive  to  any  instances  of  poor  writing.  Writing 
style  may  be  a  significant  overall  issue  with  regard  to  a  particular  piece  of  software,  but  minor  writing  errors 
should  not  be  allowed  to  occupy  the  energy  of  a  formal  group  review.  The  use  of  automated  spelling  and  style 
checkers  by  the  author  can  be  very  helpful. 

The  attached  handout,  “Software  Specifications:  Categories  for  Remarks”  (see  p.  66)  lists  examples  of  helpful 
types  of  issues. 

The  actual  use  of  software  reviewing  in  the  practice  of  software  development  is  quite  varied.  IBM  makes 
extensive  use  of  “Software  Inspections,”  and  Bell  Labs  is  noted  for  rigorous  walk-throughs.  In  general,  our 
graduates  arc  likely  to  see  lots  of  informal  walk-throughs,  frequently  involving  only  one  or  two  persons,  or 
members  of  the  project  team,  which  is  not  quite  what  the  idea  of  group  review  by  peers  from  outside  the  project 
is  all  about.  Technical  reviews  may  consist  of  a  formal  acceptance  procedure  as  part  of  a  “configuration 
management”  process,  with  an  individual  or  a  fomial  committee  which  is  charged  with  assuring  the  quality  of 
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pu'cos  ol  strllwua'  Iwloiv  they  arc  accopied  by  technical  services  people.  The  following  list  contains  a  number 
of  s|Hvilic  iKiu'liis  which  have  Ivon  uttiibiitcd  to  .software  reviewing. 

•  Koduction  in  cinuN  m  il\c  llrsi  pixnluction  run  of  a  one-line  maintenance  change  (55%  ->  2%, 

(1  rejoilmanO^I). 

•  Reduction  in  crnirs  in  first  paiduciion  riin>  r  iTr  linteniuicc  changes  (85%  ->  20%,  [Froedman82]). 

•  Reduction  in  seriousness  of  cnors  (a'lM,''  e.) ,  ff  iyedman82]). 

•  Reduction  in  pauluction  cnisne;,  i.mi  ■,  ..  si  six  months  of  operation  of  a  system  (77%, 

(l-roodntan821). 

•  Reduction  of  83%  in  Held  trouble  rci>:r.s  (f  '•e3dman82]. 

•  Reiluction  of  errors  in  COBOL  c'xic  (fron  5.5  eiTors/thousand  lines  to  1.0  errors/thousand  lines, 
(Freodman82p, 

•  Increu.sed  productivity  through  early  detection  and  removal  of  errors,  and  defect  prevention 
([Fre0dman82].  and  statistics  in  [Boehm81]). 

•  l  inding  and  removal  of  .10%  to  70%  of  logic  and  design  errors  in  typical  programs,  as  high  as  80% 
ol  pmgrain  errors  (IBM,  (Myers79]). 

•  Higher  test  scores  from  students  who  use  peer  group  review  techniques  on  each  other’s  code  in 
programming  classes  [L8nx)378,  Shelly82]. 
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Notes  on  the  Software  Development  Life  Cycle 


The  development,  installation,  and  modification  of  many  software-supported  systems  involves  significant  human 
effort,  and  defects  in  software  can  have  serious  consequences.  In  all  but  the  most  trivial  software  (software 
researchers  use  the  term  “toy  programs”),  identifiable  types  of  recurring  activities  occur.  The  most  common  of 
these  software  development  activities  are  finequently  named  and  studied  together  in  the  context  of  software  life 
cycle  models. 

A  list  of  software  development  tasks  which  are  often  included  in  software  life  cycle  models  is  attached  to  these 
notes  (p.  10).  The  degree  to  which  this  list  captures  what  software  developers  actually  do  or  should  do  depends 
on  three  major  factors:  the  size  of  the  project,  the  application  area,  and  the  development  context  Some  activities 
may  be  omitted  or  involve  minor  effort;  others  may  be  done  in  parallel  or  in  different  sequence  than  what  is 
list^;  and  still  others  may  have  to  be  redone  as  subsequent  system  development  effort  provides  additional  input. 

The  descriptiorrs  of  the  items  in  the  list  reflect  the  fundamental  importance  of  the  concept  of  abstraction  to 
successful  software  development.  Abstraction  is  simply  the  suppression  of  unnecessary  detail.  Software  : 
terns  consist  of  layer  upon  layer  of  abstraction,  including  macliine  code,  source  code,  operating  systems,  job 
control  statements,  and  events  in  the  application  domain.  Abstraction  allows  a  complex  task  to  be  worked  with 
in  simpler  chunks:  it  is  the  essence  of  the  meaning  of  the  word  structured  in  the  term  stmctured  programming. 

In  ihe  context  of  Computer  Science  319,  Software  Engineering  Principles,  the  items  in  this  list  of  software 
development  activities  provide  a  framework  of  things  which  must  be  interrsively  smdied.  Concepts  of  software 
quality  assurance  will  be  integrated  throughout  the  course,  with  the  exception  of  program  testing,  which  is 
viewed  as  an  implementation  concern.  The  process  of  modifying  existing  software  (i.e.,  software  maintenance) 
is  viewed  as  a  special  case  of  software  development,  since  all  of  the  listed  software  development  activities  can 
be  considered  separately  for  software  maintenance  tasks.  The  essential  maintenance  concept  is  tiiat  it  is  of 
overwhelming  importance,  and  all  software  development  activities  must  be  designed  to  facilitate  later  modifica¬ 
tions  to  software  development  products. 

The  concept  of  software  prototyping  will  be  treated  separately  because  it  incorporates  much  of  the  requirements- 
specification-design-implementation  sequence  of  activities  into  a  single  activity.  One  use  of  prototyping  is  to 
provide  an  executable  model  which  illustrates  what  might  be  done  to  serve  user  needs,  and  to  a  certain  extent 
how  these  system  behaviors  might  be  implemented.  The  goal  of  a  prototype  is  to  illustrate  a  subset  of  system 
behaviors  and  functions  at  the  user  interface  level,  while  avoiding  implementation  details  as  much  as  is  practi¬ 
cally  possible,  prototyping  is  especially  helpful  in  demonstrating  system  behaviors  which  are  new  to  the  user. 
A  flexible  prototype  iso  allows  system  engineers  and  users  to  develop  their  understanding  of  what  is  needed 
and  why.  Prototypes  are  weakest  at  demonstrating  perfonnance  characteristics  of  proposed  systems  (e  g.,  re¬ 
sponse  time  and  interaction  with  existing  systems). 

The  principal  difficulty  with  software  development  and  quality  assurance  is  the  dramatic  increase  of  the  diffi¬ 
culty  of  software  development  projects  as  they  become  more  complex.  In  other  words,  when  a  task  involves  too 
many  considerations,  it  requires  an  exorbitant  effort  and  it  involves  a  high  risk  of  error.  The  hierarchical 
abstractions  represented  by  the  requirements-specification-design-implemcntation  approach  can  be  helpful  be¬ 
cause  they  structure  the  various  types  of  relevant  details  into  separate,  distinguishable  activities.  This  approach 
also  facilitates  two  techniques  for  managing  complexity:  projection  and  partitioning.  Projection  in  software 
development  is  the  principle  of  simplifying  a  task  by  viewing  it  from  different  perspectives.  For  example,  a 
library  system  can  be  viewed  from  the  perspective  of  patrons,  circulation,  acquisitions,  and  cataloging.  Par¬ 
titioning  is  the  principle  of  divide  and  conquer.  For  example,  software  which  performs  data  entry  and  validation 
functions  can  often  be  developed  separately  from  software  which  performs  specific  data  processing  functions. 

If  a  software  development  activity  appears  to  overlap  two  different  life  cycle  phases,  then  the  principle  of 
abstraction  is  being  violated  by  including  unnecessary  detail.  This  increases  the  complexity  of  that  particular 
activity,  and  any  future  efforts  to  reconsider  it,  especially  during  system  maintenance.  Whenever  system  com¬ 
plexity  is  a  dominant  concern,  which  it  often  is  in  practice,  software  life  cycle  models  provide  essential  guide¬ 
lines  for  an  organized  approach  to  software  development. 

Life  cycle  models  also  help  to  keep  priorities  in  order.  Consider  the  popular  remark  that  “When  you’re  up  to 
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your  bleep  in  alligators,  it’s  hard  to  remember  that  your  original  objective  was  to  drain  the  swamp.”  In  terms  of 
a  system  life  cycle  model,  draining  the  swamp  is  a  system  requirement,  the  system  specification  includes  ditches 
specific  drainage  rates,  the  design  states  where  the  ditches  will  go,  and  the  alligators  are  an  implementation 
1-  jlem. 

In  the  “real  world”  practice  of  software  development,  a  life  cycle  model  provides  a  useful  way  of  planning, 
organizing,  controlling,  directing,  and  evaluating  software  development  activities.  In  short,  it  supports  the  man¬ 
agement  of  software  development  projects.  Written  records  document  the  results  of  software  development 
activities.  The  adoption  of  a  particular  life  cycle  model  implicitly  defines  appropriate  types  of  software  docu¬ 
mentation,  together  with  an  appropriate  level  of  abstraction  for  each,  and  the  audiences  that  this  document  must 
communicate  with.  For  example,  a  software  system  specification  must  be  understood  by  the  authors  of  the 
software  requirements,  so  that  they  can  verify  the  product’s  completeness  and  consistency  with  their  intent  The 
specification  also  must  be  understood  by  user  clientele  who  are  part  of  a  particular  approval  process;  it  must 
state  the  external  behaviors  of  the  intended  system;  and  it  must  provide  adequate  direction  to  the  system  desig¬ 
ners  who  will  woric  from  it. 

The  amount  and  form  of  documentation  is  often  subject  to  local  standards  extrinsic  to  a  specific  sofn  are  devel¬ 
opment  project,  but  the  guiding  principle  for  documentation  should  be  that  it  must  be  usrful.  The  difficulty  in 
applying  this  guideline  is  that  the  authors  may  never  realize  who  will  use  the  document  and  for  what  reasons. 
This  is  why  extrinsic  standards  and  apparendy  arbitrary  guidelines  are  needed.  A  helpful  general  guideline 
should  be  to  document  each  life  cycle  activity  as  it  happens  and  have  that  documentation  reviewed  before  that 
particular  life  cycle  activity  is  considered  to  be  complete.  The  reviewing  procedure  needs  to  be  formalized, 
complete  with  sign-offs  and  management  reports,  because  of  its  importance.  It  also  needs  to  be  organized  and 
carefully  managed  because  of  the  difficulty  of  checking  the  correctness  of  something  which  cannot  be  “tried 
out”  and  fiddled  with. 

Because  the  process  of  software  development  is  dynamic  and  context-dependent,  additional  experience  with 
large  projects  is  required  in  order  to  appreciate  the  complexity  which  is  not  apparent  in  the  any  software  life 
cycle  model.  Computer  science  internships  and  Computer  Science  320,  Software  Engineering  Practice,  provide 
•  of  this  experience.  The  following  list  should  be  helpful.  Please  feel  free  to  discuss  this  handout  with  your 
ii....,uctors. 
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1 .  Requirements  Definition 

•  Goals  -  Understanding  of  what  is  needed  and  why  it  is  needed.  A  statement  of  a  requirement 
is  likely  to  have  the  form  “The  system  must  <statement  of  user  need>.”  Words  like  must, 
should  and  desirable  indicate  prioritized  requirements.  General  system  goals  and  ideal  sys¬ 
tem  behavior  may  be  included  with  statements  of  verifiable  system  requirements. 

•  Input  -  Data  collected  about  the  problem  domain,  particularly  from  any  existing  systems,  and 
the  analysis  of  that  data  by  “system  analysts.” 

•  Ouqjut  -  A  system  requirements  document,  which  states  the  system  goals  and  specific  needs 
to  the  best  ability  of  sysit  m  analysts  and  users  (or  their  representatives).  The  rationale  for 
specific  requirements  i 'd  a  model  of  pertinent  features  of  the  application  context  should  be 
included  with  the  requirements  document  when  they  are  appropriate  for  the  needs  of  a  par¬ 
ticular  project 

»  Control  -  Approval  of  users  or  their  representatives,  formal  reviews  of  the  requirements  and 
derived  software  elements,  especially  a  software  system  specification  document  and  system 
validation  testing. 

2.  System  Requirements  Specification 

•  Goals  -  a  statement  of  what  a  system  that  will  meet  the  stated  requirements  will  look  like  to 
the  user  (i.e.,  system  behaviors  at  user  interfaces).  A  system  specification  is  stated  as  a 
verifiable  system  behavior  that  the  “target  system”  will  have.  (Note  :  the  writing  of  detailed 
“design  specifications”  which  describe  the  operating  constraints  and  behaviors  of  individual 
system  modules  is  part  of  “system  design.”) 

•  Input  -  System  requirements,  knowledge  of  the  capabilities  of  the  specified  computing  envi¬ 
ronment  for  the  target  system,  and  expertise  in  building  computer-based  systems. 

•  Output  -  A  document  which  states  all  relevant  system  behaviors  and  constraints,  and  verifi¬ 
cation  procedures. 

•  Control  -  Approval  procedures  and  system  testing. 

3.  System  Design  (High-level  design) 

•  Goals  -  A  statement  of  how  the  specifications  will  be  met. 

•  Input  -  System  specifications  and  system  building  cxperii.se. 

•  Output  -  System  architecture  and  module  specifications  that  are  below  the  level  of  user 
interfaces. 

•  Control  -  Approval  procedures,  module  testing,  system  testing. 

4.  Implementation  (Low-level  design) 

•  Goals  -  Program  design,  coding,  testing,  and  debugging,  together  with  system  constmetion 
and  certification. 

•  Input  -  System  architecture,  functional  specifications  for  programs,  system  data  specifica¬ 
tions,  module  interface  specifications,  system  and  program  constraints. 

•  Output  ■  A  finished  product. 

•  Control  -  Acceptance  procedures  and  criteria. 
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5.  Installation 

•  Goals  -  Training  of  users  and  operators  and  initiation  of  system  operation. 

•  Input  -  System  and  user  documentation,  executable  code,  and  an  application  context. 

•  Output  -  A  smoothly  functioning  sy.stem  which  meets  user  requirements. 

•  Control  -  Acceptance  testing,  and  post-installation  evaluation. 
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Sample  Software  Defects 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Examples  of  defects  that  might  be  encountered  in  a  software  technical  review  and  how  a  review 
team  might  document  them. 

Objectives:  To  be  helpful,  sample  defects  should  enable  students  to  accomplish  the  following  objectives: 

•  Learn  what  characteristics  experts  with  experience  and  foresight  consider  to  be  defects. 

•  Identify  similar  defects  in  software  artifacts. 

•  Document  the  defects  which  they  detect  in  technical  reviews 

•  Make  an  effort  to  avoid  injecting  similar  defects  into  software  which  they  write. 

Prerequisites:  Students  must  understand  relevant  technical  concerns  for  the  type  of  software  in  which  each 
sample  might  be  found,  the  application  domain  of  the  sample,  and  the  specific  goals  of  the  sample  software. 

Procedure:  The  samples  may  be  used  as  lecture  aids  or  as  illustrative  examples  in  textual  materials  given  to 
students. 

Evaluation:  Students  may  graded  on  their  ability  to  detect  similar  defects  and  produce  similar  documen¬ 
tation.  A  software  technical  reviewing  project  may  be  used  to  evaluate  student  accomplishment  of  the  above 
objectives  in  a  specific  context. 

Commentary:  One  of  the  diffioilties  associated  with  detecting  defects  in  technical  reviews  is  the  need  to 
consider  the  details  of  a  piece  of  software  that  is  only  a  part  of  an  integrated  set  of  software  artifacts.  This 
software  in  turn  constitutes  the  implementation  detail  of  a  software  development  project.  It  is  difficult  to 
provide  short  examples  which  arc  still  meaningful  after  being  extracted  from  a  document  that  is  suitable  for 
technical  review.  The  following  examples  arc  published  in  rough  draft  form  because  fjxamples  are  essential  to 
learning  the  undcriying  concepts,  not  because  these  particular  examples  arc  exceptionally  good.  Moreover, 
technical  reviewers  carmot  limit  their  concern  to  types  of  defects  which  they  already  know.  Rather,  technical 
reviewers  must  strive  to  detect  defects  which  extend  their  knowledge  of  what  constitutes  a  defect.  (Note; 
[Winograd86]  discusses  how  language  can  limit  our  perceptions.  An  analogy  can  be  drawn  with  the  blindness 
that  can  result  from  overreliance  on  samples  of  defects  or  a  reviewer’s  checklist.) 
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Sample  Defects 


ORIGINAL  TEXT  TOLLOWS. 


Computer-Assisted  lUP  Degree  Audit:  System  Requirements  by  Joe  Smith  and  Jane  Doe,  8/25/86 


A.  OVERVIEW 

ITiis  paper  outlines  the  development  of  a  computer-based  component  of  a  system  to  provide  computer  support; 
for  lUP  student  advising.  The  computer-based  component  will  be  integrated  within  a  single  student  records 
system  serving  multiple  needs  throughout  the  academic  affairs  division.  At  the  same  time,  this  component  is  not 
an  end  in  itself;  it  must  be  integrated  into  the  larger  package  offered  in  support  of  excellence  in  all  dimensions  of 
the  lUP  quest  for  quality.  . . . 


THE  ORIGINAL  DOCUMENT  CONTAINED  EIGHT  MORE  PAGES  OF  PROSE. 


ORIGINAL  TEXT  FOLLOWS  WITH  REVIEWER  COMMENTS  IN  ITAUCS. 


Computer-Assisted  lUP  Degree  Audit:  System  Requirements  by  Joe  Smith  and  Jane  Doe,  8/25/86 


A.  OVERVIEW 

'  paper  outlines  the  development  of  a  computer-based  component  of  a  system  to  provide  computer  support 
for  1  UP  student  advising. 

15-021*  This  is  declared  to  be  a  requirements  document.  An  “outline  for  the  development  of  a 

system"  should  not  be  written  until  the  requirements  for  the.  system  are  clearly  documented 
and  agreed  upon. 

The  computer-based  component  will  be  integrated  within  a  single  student  records  system  serving  multiple  needs 
throughout  the  academic  affairs  division. 

11-04]  Cite  a  reference  which  describes  this  student  records  .system  in  detail.  A  system  which  is 

integrated  with  the  student  records  system  cannot  he  specified  until  the  student  records 
system  is  clearly  specified. 

At  the  same  time,  this  component  is  not  an  end  in  itself;  it  must  be  integrated  into  the  larger  package  offered  in 
support  of  excellence  in  all  dimensions  of  the  lUP  quest  for  quality. 

1 1 -04]  This  system  is  proposed  as  a  subsystem  of  student  records,  which  is  declared  here  to  be  a 

subsystem  of  some  unspecified  larger  system.  This  system  cannot  be  required  to  be  inte¬ 
grated  into  a  system  which  does  not  even  have  a  name,  let  alone  specific  form. 


ORIGINAL  TEXT  WITH  COMMENTS  CONTINUES. 

*The  numbers  in  these  examples  are  taken  from  the  attached  list.  See  al.so  the  section.  Categories  for  Defects  in 
Software  Technical  Reviews. 
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ERROR 

CATEGORY  PROBLEM  DESCRIPTION 


*1-00  Missing/Incomplete/Inadequate 
1  -01  Criteria  for  a  system  decision  missing  or  inadequate 

1-02  Interface  characteristics  missing 
1  -03  Accuracy  or  precision  criteria  missing 
1  -04  Description  of  context  inadequate 

1-05  Processing  specification  missing 
1  -06  Error  recovery  specification  missing 

1-07  Missing  emphasis 

1-08  Data  or  process  validation  criteria  missing 
1-09  Acceptance  criteria  missing 

1- 10  Data  specification  missing 

2- 00  Inconsistent/Incompatible 

2- 01  Two  or  more  specifications  disagree 

2-02  Incompatible  witli  existing  standards 

2- 03  Incompatible  w  ith  existing  systems 

3- 00  Unclear 

3- 01  Terms  or  acronyms  need  to  be  defined 

3-02  Ambiguous  wording 

3-03  Muddled  writing 

3- 04  Specification  doesn't  make  sense 

4- (XJ  Extra 

4- 01  Outside  of  the  scope  of  this  project 

4-02  Unnecessary  detail 

4-03  Redundant  or  wordy 

4- 04  Overiy  restrictive  (includes  specirications  which 

are  stated  at  too  low  a  level) 

5- 00  Incorrect  form 

5- 01  Typographical  or  spelling  error 

5-02  Writing  error 

5-03  Word  processing  error 

5- 04  Violation  of  standards 

6- 00  Incorrect  technical  detail 

6- 01  Specified  processing  inaccurate  or  imprecise 

6-02  Specified  processing  inefficient 

6-03  Specification  not  testable 

6-04  Specification  not  modifiable 

6-05  Ctescription  of  problem  or  context  incorrect 
6-06  Teclinical  error 

7- 00  General  remarks 

*  Derived  from  a  list  in  Bell,  T.  E.,  and  Thayer,  T.  A.,  “Software  Specifications:  Are  They  a  FToblem?”  Proc. 
IEEE! ACM  Second  Inti.  Cortf.  on  Software  Engineering,  October  1976. 
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Questions  for  Knowledge  Assessment 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Pretest  of  student  understanding  of  software  development  phases  and  skill  in  writing  software 
development  prose. 

Objectives:  To  assess  student  knowledge  and  skill.  The  results  can  be  used  to  adjust  course  content  to  meet  the 
needs  of  the  students. 

Procedure:  These  questions  should  be  used  before  students  begin  studying  the  material  in  CM-3. 

Evaluation:  Answers  and  comments  are  printed  in  italics. 

Commentary:  Students  must  understand  the  concept  of  software  development  phases  before  they  can  be  com¬ 
petent  technical  reviewers  of  software.  The  questions  provide  practice  for  students  and  knowledge  assessment 
for  their  instructor.  Although  the  questions  have  not  yet  been  used  for  homewo±  exercises  or  examinations, 
such  use  appears  to  be  viable.  The  author  welcomes  comments  from  users  of  these  materials. 
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and  Scoring  Guidelines 


General  directions:  Complete  each  item  to  the  best  of  your  ability.  Your  instructor  is  trying  to  determine  how 
much  you  know  in  order  improve  the  teaching  of  this  course. 

Part  I.  Identification 

Directions:  Label  each  item  as  an  example  of  one  of  the  following: 

•  a  system  requirement  (REQ) 

•  a  system  specification  (SPEQ 

•  a  design  detail  (DES) 

•  none  of  the  above  (NA) 

Include  any  comments  that  might  help  the  instructor  develop  an  understanding  of  what  you  know. 

1 .  For  an  electronic  mail  system:  The  electronic  mail  system  will  be  able  to  distribute  a  single  message  to  up  to 
32  destinations  with  a  single  command.  <SPEC.  The  phrase  "will  be  able  to”  is  characteristic  of  SPEC s.  This 
statement  expresses  a  specific  system  capability,  not  a  user  need.> 

2.  For  your  dream  house:  All  the  living  areas  of  the  house  must  have  connections  for  telephones  and  cable  TV, 
<REQ.  The  phrase  "must  have"  is  characteristic  of  REQs.  A  specification  would  say  what  kind  of  wiring, 
where  the  modular  connections  will  be,  what  specific  kind  of  connections,  etc.> 

3.  For  a  text  editor  The  virgule  (/)  will  be  used  as  a  standard  string  delimiter.  <DES.  A  specification  might 
state  that  a  specific  symbol  will  be  used  as  a  string  delimiter,  while  the  choice  of  that  character  is  generally  a 
design  decisions 

4.  For  an  online  grade  book:  Students  must  be  able  to  check  their  grades  by  interacting  directly  with  a  host 
computer  system.  <REQ.  This  statement  describes  a  capability  that  the  system  must  have.> 

Part  n.  Reviewing  Software  Requirements 

Directions:  Qassify  each  of  the  following  statements  of  software  requirements  as  follows: 

•  acceptable  as  is  (OK) 

•  needs  minor  revision  (OKR) 

•  needs  a  complete  rewrite  (NOK) 

Briefly  explain  why  you  decide  on  any  OKR  or  NOK  re.spon.se. 

5.  For  an  appointment  scheduling  system:  The  system  must  be  able  to  automatically  schedule  a  meeting  which 
does  not  conflict  with  the  time  schedules  of  its  participants.  <OKR.  The  reasons  for  the  OKR  are  constraint, 
preference,  and  exception  handling  (which  might  be  addressed  t.isewhere  in  a  specific  software  requirements 
document).  This  statement  must  be  followed  by  a  statement  about  constraints  and  preferences  and  what  will  he 
done  when  no  solutions  are  possible.  > 

6.  For  a  text  editor:  The  system  must  be  able  to  find  and  display  aU  instances  of  a  user-specified  character  string 
in  a  text  file.  The  display  must  indicate  where  the  string  occurs  within  the  text  file.  <0K.> 

1.  For  an  online  university  student  database:  The  system  will  store  student  identification  numbers  as  packed- 
decimal  integers.  <NOK.  This  is  a  design  detail;  it  is  entirely  out  of  place  in  a  software  requirements 
document. > 

Part  m.  Reviewing  a  Software  Specification 

Directions:  Gassify  the  usefulness  of  each  of  the  following  software  specification  statements  using  the  same 
categories  with  which  you  responded  to  the  items  in  Part  II.  These  statements  are  not  necessarily  connected  with 
questions  5-7. 

8.  For  an  appointment  scheduling  system:  The  system  will  allow  an  authorized  user  to  list  all  possible  times  for 
a  meeting  and  then  select  one.  <NOK.  A  slight  case  of  ambiguity:  does  the  system  pick  one,  or  does  the  system 
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user  pick  one?> 

9.  ^•'r  a  text  editor;  The  system  will  have  “global  edit”  capability;  that  is,  a  user  may  replace  or  remove  all 
oc  mces  of  a  given  string  within  a  single  document  with  a  single  command.  <OK.> 

10.  For  an  online  gradebook:  Each  course  may  have  up  to  32  sections,  and  up  to  128  students  per  section. 
<OK.> 

Part  IV.  Open-Ended  Responses 

11.  Directions:  For  any  two  of  the  following  compute.r-.supported  systems,  write  a  one-sentence  statement  of  a 
software  REQ  and  a  second  sentence  which  states  a  related  software  SPEC. 

•  An  online  library  catalog  system  for  books  (that  is,  a  system  to  replace  tlte  card  catalog) 

•  An  electronic  personal  planning  and  appointment  calendar 

•  An  online  university  course  registration  system 

•  A  medical  diagnostic  system 

•  A  word  processor 

Sample  answers: 

REQ:  The  online  catalog  system  must  provide  all  the  functional  capabilities  of  the  current  card  catalog. 

SPEC:  A  patron  will  be  able  to  search  for  a  book  by  title  or  author. 

REQ:  The  electronic  calendar  must  be  able  to  determine  all  possible  times  to  schedule  a  meeting  that  does  not 
conflict  with  the  personal  time  schedules  of  any  of  the  meeting  participants.  SPEC:  The  electronic  calendar 
system  will  manage  the  personal  appointments  of  up  to  512  users. 

12.  Directions:  For  any  two  of  the  following  software  system  needs,  write  a  software  REQ,  a  related  SPEC,  and 
a  DES  response. 

o  An  operating  system  JCL 

•  An  electronic  mail  system 

n  online  university  registration  system 

Sample  answers: 

For  an  operating  system  JCL: 

REQ:  The  user  interface  to  the  operating  system  must  be  user-friendly.  <NOK:  Goals  are  OK  in  requirements 
statements,  but  BE  SPECIFIC .> 

SPEC:  The  name  for  a  system  command  may  he  abbreviated  by  any  of  a  set  of  letters  which  agrees  with  the 
initial  letters  in  exactly  one  valid  command. 

DES:  The  high-level  command  language  parser  will  consist  of  a  command  identifier  with  appropriate  error - 
handling,  together  with  links  to  corresponding  command  processors. 

For  an  electronic  mail  system: 

REQ:  The  system  should  allow  mail  to  be  addressed  by  name. 

SPEC:  The  system  will  be  available  24  hours  a  day,  7  days  a  week.  <OKR:  This  specification  must  be 
qualified — naivety — by  saying  "whenever  the  host  computer  is  up."  A  more  detailed  response  might  state  a 
performance  level,  with  alternatives  to  the  mail  system  when  the  system  is  not  available.  > 

DES:  System  users  will  communicate  to  the  mail  system  by  keying  a  one-digit  command  number  which  cor¬ 
responds  to  a  context-dependent  menu  of  valid  mail  commands. 

For  an  online  university  registration  system; 

REQ  ■  The  system  must  automatically  .satisfy  all  reasonable  registration  requests  from  students.  <NOK:  vague, 
not^  ble.> 

SPEC:  The  system  will  reject  a  registration  request  for  which  a  student  does  not  meet  all  the  prerequisites.  This 
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restriction  may  be  overridden  by  the  course  instructor  keying  in  a  special  permission  for  the  student. 


DES.  In  order  to  guard  against  fraudulent  registration,  students 
an  account  that  matches  their  current  student  id. 


may  only  register  when  they  are  logged  on  with 
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Guidelines  for  Teaching  Concepts 

John  A,  Cross 

Indiana  University  of  Pennsylvania 


The  following  guidelines  were  developed  at  Indiana  University  of  Pennsylvania  during  the  teaching  of  the 
course.  Software  Engineering  Concepts. 


•  Practice:  Appropriate  practice  activity  is  essential  to  learning  software  technical  reviewing  proce¬ 
dures  and  the  underlying  theory. 

•  Background  Knowledge:  The  successful  implementation  of  any  software  reviewing  activity  re¬ 
quires  that  the  reviewers  have  adequate  knowledge  of  goals  and  guidelines  for  the  software  artifact 
they  are  reviewing.  Reviewers  need  background  knowledge  of  both  the  application  domain  and  tlie 
particular  software  element  under  review;  and  tliese  needs  must  be  addressed  by  the  instructor  in  a 
manner  that  is  suitable  to  the  goals  of  each  particular  software  reviewing  activity. 

*  Documents:  The  need  for  background  knowledge  about  the  application  domain  requires  that 
reviewers  have  access  to  the  documents  that  are  used  to  prepare  the  particular  software  ar¬ 
tifact.  In  many  situations,  these  documents  are  the  principal  means  of  communicating 
application-specific  domain  knowledge  to  reviewers.  Thus,  reviewers  should  have,  for  ex¬ 
ample,  a  specification  and/or  a  design  document  for  a  program  before  they  attempt  to  review 
the  code.  A  viable  supplemental  or  alternative  procedure  to  providing  documents  to  students 
is  to  have  an  applicadon-domain  expert  available  for  consultation  by  the  reviewers. 

•  Guidelines:  The  reviewers  also  need  guidelines  for  software  documents.  Fagan  [Fagan86] 
calls  these  guidelines  exit  criteria.  The  instructor  can  communicate  these  guidelines  through  a 
combination  of  class  presentations,  printed  guidelines  for  specific  types  of  software  artifacts, 
and  checklists. 

•  Goals  &  Procedures:  Students  must  understand  goals  and  procedures  for  software  reviewing  be¬ 
fore  they  can  participate  in  a  software  reviewing  activity.  Classroom  discussion  should  include: 
what  reviewers  should  search  for,  how  they  should  search,  how  they  should  report  their  findings, 
and  how  they  should  work  as  a  group.  The  support  materials  in  other  sections  of  this  document 
provide  some  examples  of  the  things  reviewers  should  report.  Instructors  may  need  to  develop 
additional  examples  that  are  similar  to  the  specific  tasks  they  have  planned  for  their  students.  In- 
stmetors  should  also  thoroughly  explain  all  procedures  for  recording  remarks,  interacting  as  a 
groun,  and  reporting  results.  If  computer-based  procedures  are  used,  instructors  will  need  to  verify 
that  all  students  are  literate  in  these  procedures. 
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Instructor’s  Checklist  for  Planning 
a  Software  Technical  Review 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Checklist  to  assist  an  instructor  in  preparing  for  a  software  technical  reviewing  activity  by  stu¬ 
dents. 

Prerequisites:  The  user  of  this  checklist  should  have  a  good  understanding  of  the  concepts  documented  in 
CM-3  and  should  have  these  support  materials  (SM-3)  available  for  reference. 

Procedures:  The  instructor  should  complete  each  item  in  the  list  before  distributing  materials  to  students  to  be 
reviewed,  providing  comments  and  document  names  were  appropriate.  The  one  exception  may  be  item  (8). 
Tliis  checklist  has  been  found  to  be  a  useful  planning  guide.  Software  developers  and  educators  believe  that 
software  technical  reviewing  procedures  must  be  monitored  constantly.  This  checklist  should  be  viewed  as  an 
important  part  of  that  process. 

Commentary:  The  author  would  appreciate  hearing  about  any  additions  instmctors  have  made  to  this  list,  as 
well  as  receiving  comments  on  its  use. 
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Directions:  Each  of  the  following  concerns  must  be  addressed  by  the  organizer  of  a  classroom  software  review¬ 
ing  activity.  The  instructor  may  find  it  helpful  to  write  specific  statements  or  references  to  the  documents  that 
satisfy  each  item.  References  in  square  brackets  refer  to  pages  in  these  support  materials  that  address  each 
checklist  item  in  a  generic  way. 


) .  Activity  ID  (Name  of  task  or  type  of  material  to  be  reviewed,  and  date). 

2.  Goals  [4,  19] 

3.  Inspectable  materials  [39] 

4.  Student  knowledge  of  guidelines  for  type  of  software  artifact  to  be  reviewed  [15] 

5.  Student  knowledge  of  application  domain 

6.  Student  knowledge  of  how  to  detect  defects  [  12, 64] 

7.  Individual  reporting  procedures  [481 

8.  Group  interaction  and  reporting  procedures  [48] 

9.  As,signment  of  individuals  to  groups,  and  roles  and  responsibilities  to  group  members  [28, 48] 

10.  Forms  [24,  30,  36,61,64) 

1 1  Grading  Guidelines  [33] 

12.  Schedule 
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Biographical  Data  Collection  Form 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Form  and  instructions  for  the  collection  of  “biographical”  background  data  about  students  in  a 
software  engineering  course. 

Objectives:  Possible  purposes  forgathering  data  about  students  include  the  following: 

1.  Getting  to  know  students:  their  abilities,  needs,  and  expectations. 

2.  Correlating  student  behaviors  or  performance  with  biographical  data. 

3.  Providing  a  basis  for  dividing  a  group  of  students  into  subgroups  or  assigning  roles  in  group 
projects.  The  sample  form  that  follows  was  designed  with  this  objective  in  mind. 

Procedures  for  Instructor:  Rationale  and  directions  for  each  item  in  the  form  are  presented  below.  Item  (2)  is 
the  only  item  which  is  not  generic,  but  the  significance  of  each  item  must  be  considered  in  the  context  of  the 
instructor’s  purpose:  the  selection  of  data  items  and  categories  for  each  item  are  context-dependent  concerns. 
(Refer  to  “An  Algorithm  for  Dividing  Students  Into  Groups,”  p.  28  for  a  detailed  procedure  for  dividing  a  class 
of  students  into  groups  for  a  software  reviewing  activity  based  on  information  obtained  with  this  form.)  It  is 
likely  that  the  data  on  this  form  will  have  greater  value  to  an  instructor  if  they  are  put  into  machine-readable 
form.  Individual  instructors  may  devise  keying  specifications  for  their  form,  or  implement  an  online  data  entry 
program. 

The  following  directions  and  rationale  are  keyed  to  the  numbered  lines  in  the  sample  form. 

1.  Some  sort  of  identification  is  required,  so  that  the  instructor  can  communicate  with  students.  It  is 
recommended  that  students  identify  themselves  by  name  on  this  form  because  it  allows  for  subjec¬ 
tive  judgments  about  which  students  woric  together  well  in  groups.  The  computer-id  may  be  useful 
in  communicating  electronically  with  students,  or  allowing  the  names  of  students  to  be  hidden 
during  a  pass  at  dividing  the  students  into  groups,  thus  preventing  subjective  judgments  from 
influencing  decisions  at  any  particular  point.  Groups  seem  to  work  better  if  they  include  both  men 
and  women,  so  a  “Sex”  item  is  included  on  the  form.  With  relatively  .small  classes,  “Student 
Name”  is  usually  adequate  to  determine  sex.  The  “Systcm-ID”  can  be  used  in  a  number  of 
helpful  ways: 

•  To  cross-reference  the  background  data  fn  additional  data. 

•  To  provide  a  unique  ID  when  names  arc  inadequate. 

•  To  provide  an  impersonal  identifier  for  each  individual’s  data  when  objectivity  or  confiden¬ 
tiality  is  desired.  In  some  institutions,  it  may  be  necessary  to  generate  such  an  identifier. 


2  The  specific  responses  to  this  question  indicate  the  knowledge  and  points  of  view  which  an  indi¬ 
vidual  might  bring  to  a  software  reviewing  task. 


3.  The  instmetor  can  adjust  the  ranges  and  the  way  the  question  is  asked  to  suit  the  nature  of  the 
students  in  the  class. 


4.  Students  may  need  to  be  told  that  an  incorrect  statement  to  this  question  may  result  in  poor  deci¬ 
sions  about  who  is  in  which  group.  Note  that  the  number  of  credits,  specific  courses,  and  grade- 
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point  averages  work  together  as  predictors  of  the  performance  of  students  in  a  software  reviewing 
activity. 


5.  The  notes  for  3  and  4  apply  to  this  item  and  the  next.  This  question  addresses  the  need  to  deter¬ 
mine  the  technical  ability  of  the  student  to  review  a  piece  of  software. 

6.  See  the  comments  for  item  5. 


7.  A  mix  of  different  ages  is  likely  to  provide  a  better  group  review  than  a  homogeneous  group.  The 
age  groups  can  be  adjusted  to  suit  the  students. 


8,  Outside  responsibilities  can  be  a  major  factor  in  limiting  the  effectiveness  of  an  individual  student 
in  a  group  project.  Adjust  this  question,  or  ask  an  additional  question  about  distractions,  depending 
on  the  students. 


9.  The  previous  note  also  applies  to  this  item. 

10.  Geographical  distribution  of  students,  commuting  factors,  and  experience  can  all  be  important 
factors.  Questions  about  these  and  other  factors  may  be  added  to  the  form  at  the  discretion  of  the 
instmctor.  Note  that  application  domain  knowledge  is  especially  helpful  in  this  context. 


Procedures  for  Students:  Students  may  complete  the  data  collection  form  in  about  ten  minutes  of  class  time.  If 
these  data  are  to  be  used  for  grouping  decisions,  reflect  that  need  in  your  planning.  Students  should  talk  with 
h  other  or  bring  questions  to  the  instructor,  so  that  any  difficulties  can  be  resolved.  Note  that  this  type  of  data 
»  also  oe  gathered  by  an  online  system.  However,  allowing  students  to  take  a  form  home  and  submit  it  later 
invites  hassles  and  incomplete  data. 

Evaluation:  The  performance  level  of  individual  students  on  software  technical  reviews  caii  be  predicted  witli  a 
calculation  such  as  the  following.  A  nearly  identical  formula  was  90%  accurate  at  distinguishing  high,  average, 
and  low  performers  for  three  iterations  of  the  activity  described  in  these  support  materials  (with  65  students). 
However,  the  formula  may  be  dependent  on  context,  so  use  it  cautiously  (see  p.  28).  Note  also  that  this  formula 
predicts  individual  performance  at  detecting  defects,  not  group  performance. 

Each  blank  in  items  3  through  6  is  assigned  a  value  of  1  through  5. 

prediction  =  1  for  each  blank  checked  in  item  2  (2  for  intcms)/2 
+  item  3  *  (item  4/5) 

-f  item  5  *  (item  6/5) 

-  (item  8  +  item  9)/10 

For  example,  consider  the  following  coded  data  for  a  24-year-old  female  student  witli  File  Processing  and 
Internship,  94  credits  with  2.9  GPA,  24  CS  credits  with  3.4  GPA,  working  12  hours  per  week,  and  taking  14 
credits. 

Mary  Smith  CKRLWPF  1(X)1(X)  4344212014 

prediction  =  (1  [for  file  processing]  2  [for  intemshiplW 
-t-  4*(3/5)  (for  overall  course  work) 

-t-  4'*'(4/5)  (for  computer  science  course  work) 

-  (12  -H  14)/10  (for  distractions) 
prediction  =  4.1 
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Note:  Sex  and  age  codes  are  not  used  to  predict  individi  )erfonnance,  but  they  are  used  to  measure  how  well 
group  members  complement  one  another. 

If  a  statistical  package  such  as  SPSSX  is  available,  the  ^graphical  data  can  be  used  for  analyses  such  as  the 
following: 

1.  Compare  predicted  with  actual  performance. 

2.  Examine  alternative  predictors  of  individual  performance. 

3.  Analyze  the  data  for  correlations  between  background  data  and  individual  ability  to  detect  specific 
types  of  defects  in  a  software  technical  review. 

Instructors  should  check  prediction  against  actual  performance  and  modify  future  procedures  in  light  of  this 
feedback. 

Commentary:  Every  item  in  this  background  data  collection  form  may  need  to  be  adjusted  to  suit  the  context  in 
which  it  is  used.  If  such  data  will  be  used  to  predict  student  performance,  the  user  must  keep  in  mind  that  the 
following  factors  are  all  fundamental  to  predicting  the  performance  of  an  individual  on  a  software  technical 
review. 

1.  Technical  ability  relative  to  tire  particular  type  of  software  artifact  under  review. 

2.  Knowledge  of  the  particular  application  domain. 

3.  Intensity  with  which  the  reviewer  is  involved  in  the  technical  reviewing  activity. 

4.  Personality  and  leadership. 

The  data  in  the  attached  form  can  be  supplemented  with  “psychological  data”  on  cognitive  style  or  personality 
type.  A  student  of  the  author  did  this.  Tlie  grouping  decisions  prompted  by  the  psychological  data  supported  the 
validity  of  the  decisions  based  on  the  biographical  data,  but  they  did  not  provide  any  clear  improvements. 
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SURVEY  OF  STUDENT  BACKGROUND 


1)  Name _ System-ID _ Sex  (M  or  F) _ 

2)  Please  use  a  check  mark  to  indicate  the  courses  which  you  have  taken  (or  transferred),  and  use  an  asterisk  (*) 
for  those  which  you  arc  taking  this  semester. 

_ File  Processing _ Database _ Operating  Systems 

_ Internship _ System  Analysis _ System  Design 

3)  Check  the  appropriate  category  for  approximate  credits  beyond  high  school. 

_ 0-30 _ 31-60 _ 61-90 _ 91-120 _ >  120  credits 


4)  Check  the  appropriate  category  for  approximate  overall  grade-point  average. 
<  2.0  2.0-2.49 _ 2.50-2.99 _ 3.00-3.49 _ >  3.49 


5)  Check  the  appropriate  category  for  total  computing  credits  completed. 
_ 0-10 _ 11-15 _ 16-20 _ 21-25 _ >25  credits 


6)  Check  the  appropriate  category  for  approximate  Computer  Science  grade-point  average. 
_ <  2.0 _ 2.0-2.49 _ 2.50-2.99 _ 3.00-3.49 _ >  3.49 


7)  Check  Utc  appropriate  category  for  your  age. 

_ 0-23  years _  24-28  years _ _  29-33  years _ >  33  years 


8)  Average  number  of  hours  you  will  be  working  (for  pay)  per  week  during  the  time  in  which  the  software 
technical  reviewing  activities  will  be  conducted:  _ . 


9)  Number  of  credits  you  are  taking  this  semester: 
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An  Algorithm  for  Dividing  Students  into  Groups 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Algoiithm  to  assign  students  to  review  groups. 

Purpose:  Obtain  balanced  groups  that  work  effectively,  without  penalizing  individuals  by  assigning  them  to 
“bad”  groups. 

Procedure:  See  algoritiim. 

Commentary;  The  task  of  dividing  students  into  groups  for  class  projects  can  be  approached  in  three  basic 
ways: 

1.  Use  a  random  selection  process. 

2.  Allow  students  to  choose  their  own  group  members. 

3.  Assign  students  to  groups  on  some  otlier  rational  basis. 

The  above  approaches  can  be  mixed.  For  example,  an  instmctor  could  pick  the  best  students  as  leaders,  then 
either  randomly  assign  the  rest  of  the  class  to  groups  or  allow  project  leaders  to  select  their  group  members  from 
the  remaining  students.  The  advantage  of  this  approach  is  basic  to  any  algorithm  an  instructor  might  use — some 
systematic  consideration  should  be  given  to  ensuring  that  each  group  has  sufficient  personnel  resources  to  com¬ 
plete  the  project  A  strictly  random  pnxedure  incorporates  a  significant  risk  of  producing  unfair  groupings 
relative  to  each  group’s  ability  to  be  successful  on  the  project.  Poor  results  might  also  occur  if  students  are 
allowed  to  group  themselves.  For  example,  the  best  students  might  try  to  band  together  or  students  might  group 
themselves  on  the  basis  of  personal  relationships  rather  than  complementary  abilities.  However,  the  instructor 
may  not  be  able  to  do  much  better,  since  individual  abilities  and  group  interactions  are  so  hard  to  predict. 

The  procedure  described  below  is  more  systematic  than  random.  It  utilizes  data  supplied  by  students,  rather  than 
relying  solely  on  the  subjective  opinions  of  the  instruaor.  The  algorithm  is  based  on  two  criteria  for  forming 
groups; 

1.  Groupings  should  be  fair;  there  should  be  an  even  balance  of  total  human  resources  for  each 
project  group. 

2.  Groupings  should  be  heterogeneous;  there  should  be  a  mix  of  different  abilities,  and  complemen¬ 
tary  skill  levels,  sex,  and  ages. 

The  problem  of  determining  groups  is  tiius  one  of  determining  abilities  and  complementary  personalities,  then 
combining  people  into  balanced  groups  which  promise  to  be  productive.  Productivity  is  a  key  concern  in  this 
context  because  productivity  may  vary  for  a  given  group  depending  on  the  task  or  the  working  environment.  The 
algorithm  given  below  has  been  used  to  group  students  for  a  software  reviewing  task;  it  may  or  may  not  work  for 
other  tasks.  However,  the  approach  represents  a  starting  point  for  a  systematic  approach  to  this  problem  in  many 
situations. 

The  following  procedure  which  has  been  automated  using  a  computer  program,  is  a  straightforward,  flexible, 
and  objective  approach  to  the  problem  of  grouping  The  procedure  is  based  on  a  biographical  data  form  (see  p. 
24)  and  a  personality  profile  form  which  attempts  to  obtain  a  rough  indication  of  the  cognitive  style  of  each 
individual.  Note  that  the  algorithm  described  below  is  but  one  solution;  other  algorithms — or  fine  tuning  of  this 
algorithm — are  also  possible. 
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The  Algorithm 


Basic  parameters:  number-of-students,  group-size 
Input  data:  Biographical  data  and  personality  profile 


1.  Compute  the  number  of  groups  so  that,  if  there  cannot  be  equal  numlx:rs  of  individuals  in  each 
group,  the  numbers  will  be  as  equal  as  possible. 

number-of-teams  -  CElL(number-of-students  /  group-size) 

2.  IDetermine  predicted  abilities  for  the  given  project  (See  p.  24.) 

3.  Son  tlie  available  individual  data  records  on  the  basis  of  predicted  abilities.  (Use  the  point  score 
for  composite  predicted  abilities.)  Assign  the  individuals  from  the  top  and  bottom  of  the  list  to  the 
same  team,  deleting  them  from  the  list,  and  repeat  this  procedure  for  each  team  until  each  team  has 
two  persons.  Examine  the  resulting  assignment  for  any  imbalance  in  the  number  of  males  on  any 
team.  If  necessary,  adjust  the  assignments  to  balance  the  number  of  males  in  each  group.  Do  the 
same  sort  of  adjustment  to  provide  heterogeneous  mixtures  of  age  levels  in  each  group. 

4.  Repeat  step  (3)  with  the  remaining  individuals,  adding  one  member  to  each  group  with  each  use  of 
the  list  of  unassigned  individuals.  With  each  new  set  of  assignments  to  groups,  adjust  the  assign¬ 
ments  to  balance  the  number  of  males  and  the  heterogeneity  of  age  brackets  of  the  people  in  each 
group. 

5.  Print  a  summary  report  of  team  demographics;  adjust  manually  if  appropriate.  Note  the  reason  for 
the  adjustment  for  possible  inclusion  in  the  algorithm. 

6.  Identify  the  group  assignments  witli  names,  and  make  any  needed  subjective  adjustments. 

7  Reprint  the  summary  report  of  team  demographics  and  repeat  steps  (6)  and  (7)  if  the  groups  are 
poorly  balanced  or  poorly  mixed, 

8.  Look  for  leadership  potential  in  each  group.  If  a  group  does  not  appear  to  have  anyone  with 
leadersliip  potential,  farther  adjustments  may  be  called  for. 

9.  Decide  whether  to  appoint  group  leaders  or  to  allow  another  form  of  leader  selection  within  each 
group. 
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Agreement  and  Release  Form 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Agreement  for  students  to  give  consent  for  participation  in  research. 

Procedure:  If  you  plan  to  use  the  data  gathered  from  classroom  assignments  for  publication  or  as  pan  of  some 
other  research,  Ac  students  should  sign  consent  forms.  The  following  form  can  be  used  to  obtain  student  permis¬ 
sion  to  collect  empirical  data. 

The  instructor  should  acquire  appropriate  administrative  approvals  of  this  form  before  using  it  (This  form  has 
been  approved  by  two  universities.)  Have  students  complete  Ae  form  and  sign  a  paper  copy  prior  to  Ae 
collection  of  empirical  data. 

Commentary:  Consent  forms  are  a  standard  part  of  studies  involving  human  subjects.  Their  use  grew  out  of  an 
eAicai  need  for  constramts  on  experiments  using  human  subjects.  Many  mstilutions  (boA  in  academia  and  in 
industry)  and  publications  require  Aem.  In  some  states  Aey  are  required  by  law.  A  general,  consent  forms 
inform  Ae  subjects  of  Ae  following: 

•  What  Aey  can  expect. 

•  What  harm  (if  any)  Aey  may  incur.  (In  this  case,  will  Aeir  participation  have  an  adverse  effect  on 
Acir  grades  or  what  they  learn  from  tlie  course?  It  should  not.) 

•  That  Aey  may  choose  not  to  participate  or  may  wiAdraw  at  any  time. 

Undergraduate  sAdents  of  Ae  auAor  responded  wiA  respect  and  interest  A  Ae  formality  of  this  procedure.  Use 
of  this  form  may  have  caused  a  healAy  "HawAome  Effect,”  m  which  sAdents  performed  better  because  Aey 
perceived  Aat  Aeir  performance  was  being  more  closely  observed  Aan  normal. 
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Agreement  and  Release  Form 


Research  Agreement  and  Release  Form 


In  this  course,  your  instructor  will  be  collecting  empirical  data  about  your  performance  on  your  assignments. 
The  special  procedures  that  this  entails  are  described  on  the  following  page.  Please  read  the  description  of  those 
procedures.  When  you  feel  that  you  imderstand  what  is  planned,  sign  the  agreement  and  release  statement. 

AGREEMENT  AND  RELEASE  STATEMENT.  By  my  signature  on  this  document,  I  signify  that  I  have  read 
and  understood  the  statement  of  planned  research.  Furthermore,  I  agree  to  participate  to  the  best  of  my  ability. 
The  data  which  are  gathered  from  this  experiment  may  be  published,  but  only  in  a  manner  which  does  not 
individually  identify  me. 


SIGNATURE 


DATE 


Mailing  address  for  a  summary  report  of  the  results  (optional): 


Questions  may  be  addressed  to  : 

<Instn'Ctor  name> 
<Addrcss> 
cTelephone  number> 
<E-mail  address> 
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Research  Procedures 


You  will  be  divided  into  groups  of  three  cr  four  students.  Grouping  will  be  based  on  your  background,  as  you 
declared  it  on  the  “Biographical  Data  Collection  Fonn."  which  you  completed  at  the  beginning  of  this  course. 
Volunteers  may  be  asked  to  serve  as  “coordinators"  and  “recorders”  for  each  group.  Each  group  will  first 
review  a  software  requirements  document  and  then  a  software  specification  of  verifiable  system  behaviors. 

The  reviewing  procedure  will  consist  of  individual  preparation  of  constructive  comments  about  the  software 
element  which  is  being  reviewed,  followed  by  a  review  group  meeting  in  which  the  group  prepares  a  written 
report  on  the  quality  of  the  software.  Two  different  procedures  for  coordinating  the  individual  preparation  of  the 
review  group  members  will  be  used.  You  will  be  graded  on  your  individual  mastery  of  the  concepts  of  software 
development  that  arc  specific  to  the  software  which  you  review  and  on  the  quality  of  your  group  review  reports. 
If  the  different  grading  procedures  result  in  significant  differences  in  your  grades,  an  adjustment  wiU  be  made  to 
assure  fairness. 

In  the  course  of  these  activities,  your  consistent,  diligent  participation  is  essential  for  our  observations  to  have 
any  value  beyond  this  course.  Therefore,  NO  LATE  ASSIGNMENTS  WILL  BE  ACCEPTED.  If  you  arc 
inadequately  prepared  for  a  review  meeting,  you  will  be  dropped  from  your  group.  If  you  miss  a  class  during  the 
period  in  which  you  are  learning  about  the  software  reviewing  tasks  or  performing  those  tasks,  you  will  not  only 
compromise  your  ability  to  score  well  in  this  course,  but  you  will  also  cause  problems  for  your  group  and  vour 
instructor.  PERFECT  ATTENDAJVJCE  AND  SINCERE  EFFORT  ON  INDEPENDENT  ASSIGNMEOTS 
WILL  BE  DEMANDED  OF  YOU. 


Alternative  to  Signing  the  Research  .Agreement  and  Release 

You  may  elect  to  not  sign  this  form.  If  you  do  not  sign  this  form,  or  if  you  fail  to  meet  group  reviewing 
responsibilities,  you  will  be  placed  in  a  review  group  with  other  students  who  are  not  part  of  the  experiment. 
However,  you  must  still  do  the  same  tasks  and  be  graded  on  the  same  performance  criteria  as  the  students  who 
participate  in  the  experiment. 


Initials  of  student: 
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Grading  Guidelines 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Topic:  Grading  student  performance  of  software  technical  reviews. 

Objectives:  These  grading  guidelines  may  serve  several  functions: 

1.  Provide  specific  direction  for  evaluating  student  performance. 

2.  Provide  cues  to  students  concerning  outcomes  that  will  be  measured  and  the  relative  importance  of 
each. 

3.  Provide  feedback  to  students  on  the  effectiveness  and  efficiency  of  both  their  individual  analysis 
and  group  interaction. 

The  specific  behavioral  objectives  that  the  grading  guidelines  address  include: 

1.  Working  individually  and  as  a  group,  students  will  produce  helpful  output  from  a  detailed  inspec¬ 
tion  of  a  piece  of  software. 

2.  Students  will  be  able  to  estimate  costs  and  benefits  in  terms  of  both  individual  and  group  effort  for 
a  software  technical  review. 

3.  Students  will  be  able  to  detect  and  document  defects  in  a  particular  type  of  software. 

4.  Students  will  introduce  fewer  defects  into  software  they  create. 

5.  Students  will  value  and  know  how  to  obtain  the  broader  understanding  that  can  be  achieved  by 
obtaining  the  views  of  different  people  and  viewing  the  software  from  different  perspectives. 

6.  Students  will  show  enthusiasm  for  the  use  of  software  technical  reviews  as  an  effective  means  to 
evaluate  the  technical  quality  of  a  piece  of  software  and  document  specific  defects. 

7.  Students  will  show  constructive  and  efficient  group  behaviors. 

Prerequisites:  The  users  of  these  grading  guidelines  need  to  know  what  constitutes  a  defect  in  the  software 
being  reviewed  and  bow  defects  are  to  be  documented.  The  software  may  include  statements  of  known  defects 
(see  “Sample  Inspection  Material,”  p.  39).  Instructors  can  depend  on  prior  knowledge  of  any  defects  that  are 
given,  of  defects  detected  by  individual  technical  review  of  the  document,  or  tliose  deliberately  inserted  into  the 
software  before  giving  it  to  the  students.  In  any  technical  review,  it  is  possible  that  previously  undocumented 
defects  will  be  reported.  If  the  .software  does  not  come  with  a  .statement  of  known  defects,  students  are  likely  to 
report  legitimate  defects  the  instructor  failed  to  notice. 

Procedure:  The  defects  documented  by  each  individual  must  be  evaluated  by  the  instructor  relative  to  how  well 
they  provide  input  to  group  interaction.  Thus,  a  copy  of  each  individual’s  annotations  must  be  submitted  for 
grading.  For  each  concern  an  individual  identifies  as  a  defect,  the  instructor  must  determine  whether  the  indi¬ 
vidual  correctly  identified  a  defect,  documented  a  spurious  concern,  or  made  an  error  in  his  or  her  understanding 
of  technical  details. 

Each  issue  must  also  be  weighted  for  the  significance  of  the  underlying  concerns  and  for  how  well  it  is  reported. 
One  approach  is  to  assign  an  integer  score  of  from  -2  to  +3  points  for  each  issue  raised.  These  scores  are  then 
summed  for  each  individual.  Since  expert  behavior  is  considered  to  be  the  correct  identification  of  roughly  30% 
of  the  actual  defects,  performance  considerably  less  than  perfect  should  be  awarded  the  highest  possible  score. 
The  author  recommends  that  (roughly) 

highest-possible-individual-score  =  0.30  *  weighted-sum-of-known-defects 

To  grade  group  performance,  the  instructor  must  compare  group  output  against  the  combined  individual  data 
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prior  to  group  interaction.  The  following  formula  can  be  used  for  grading  the  defects  reported  by  the  group. 

•  -1  to  +2  points  for  combining  similar  issues  raised  by  more  than  one  individual  in  a  group. 

•  -1  to  +2  points  for  editing  the  way  an  issue  is  reported. 

•  -3  to  +2  points  for  removing  an  issue  from  the  group  report.  The  point  value  for  this  group  action 
should  be  the  negative  of  the  point  value  for  the  underlying  defect. 

•  -1  to  +3  points  for  (synergistically)  including  an  issue  in  the  group  report  that  was  not  in  the  report 
of  any  individual  in  the  group. 

The  instructor  must  judge  the  relative  merits  of  a  particular  group  point  total  in  the  context  of  a  particular  task. 
Note  that  the  amount  of  measurable  “synergism”  in  the  group  report  is  likely  to  be  low,  even  when  there 
appears  to  be  much  group  interaction.  A  score  of  +20  to  +25  points  may  be  a  good  performance. 

The  Summary  Report  and/or  Related  Issues  Report  must  also  be  graded.  The  instructor  can  simply  assign  a 
possible  point  total  and  grade  the  summary  reporting  on  that  basis. 

A  possible  set  of  class  scores  might  look  like  this: 


Individual 

Group  No. 

Individual 
Raw  Norm. 

Group 

Raw  Norm. 

Group 

Reports 

Composite 

Score 

1 

1 

+  12 

+20 

+11 

+  16 

+6 

+42 

2 

1 

+  18 

+23 

+11 

+  16 

+6 

+45 

3 

1 

+  10 

+19 

+  11 

+  16 

+6 

+41 

4 

1 

+6 

+10 

+  11 

+  16 

+6 

+32 

Possible 

+62 

+25 

- 

+25 

+  10 

+60 

Evaluation;  The  success  of  the  grading  process  might  be  evaluated  in  terms  of  either  the  functional  or  be¬ 
havioral  objectives  stated  above.  Both  sets  of  objectives  are  important,  and  the  simplest  methods  of  evaluation 
are  the  most  appropriate: 

•  Are  these  guidelines  practical?  (They  worked  for  the  author  of  this  document.  The  experience  of 
other  users  is  now  essential.) 

•  Are  the  resulting  grades  accurate?  (Thus  far,  they  appear  to  correlate  with  student  grades  on  other 
tasks  and  in  other  courses.) 

•  Do  these  grading  procedures  foster  the  desired  student  behavior?  (Experience  with  these  procedures 
is  too  limited  to  assert  that  they  offer  significantly  better  learning  advantages.  However,  they  deal 
directly  with  tlie  details  of  student  performance  or  nonperformance,  which  is  the  essence  of  what 
should  produce  good  learning  outcomes.) 

•  Are  there  better  procedures?  (Certainly  variations  are  possible,  especially  with  the  suggested  point 
values  and  normalizing  procedure.  However,  the  details  of  determining  the  value  added  by  both 
individual  and  group  effort  must  be  considered  in  any  valid  procedure. ) 

Commentary:  The  ability  to  automatically  number,  identify  with  initials,  and  merge  student  araiotations  is  very 
helpful  in  this  procedure.  Computer  programs  to  do  this  processing  are  relatively  easy  to  constmet.  Instructors 
should  consider  a  variety  of  software  tools  and  statistical  analyses.  However,  the  following  type  of  chart  should 
be  considered  an  essential  method  of  data  presentation  for  learning  exercises  in  software  technical  reviewing 
([Myers 78]  contains  similar  charts). 
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Student  Opinion  Form 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Form  to  solicit  student  opinion  about  S('ftware  technical  review  experience  in  the  classroom 
(“debriefing”  form). 

Purpose:  Obtain  both  open-ended  and  coded  comments  from  participants  in  a  software  technical  review.  The 
form  can  be  used  to  obtain  data  from  the  perspective  of  participants  in  order  to  diagnose  problems  or  substantiate 
facts  about  the  software  technical  review  that  are  not  otherwise  documented. 

Procedure:  The  form  should  be  used  after  students  have  performed  a  software  technical  review  and  have  sub¬ 
mitted  the  gr  oup  report  Students  should  be  aware  that  their  answers  will  not  affect  their  grade. 

The  instructor  may  want  to  write  a  simple  report  to  provide  feedback  to  the  students  on  the  gist  of  their  re¬ 
sponses  on  this  form.  Public  reporting  should  never  identify  an  individual  student,  in  accordance  with  the  con¬ 
tract  made  in  the  Agreement  and  Release  Form. 

Data  gathered  from  this  form  must  never  affect  student  grades.  Siunmary  statistics  should  be  computed  for 
coded  items,  which  are  organized  in  a  way  that  makes  them  amenable  to  aniysis  by  a  statistics  package  such  as 
SPSSX.  Note  that  interrelationships  mig^t  provide  interesting  insight,  particularly  with  regard  to  group  inter¬ 
actions.  Other  data  that  can  be  linked  to  the  data  from  this  form  by  means  of  the  “System-ID”  that  also  appears 
on  other  forms  in  these  support  materials. 

Commentary:  Tlie  coded  responses  have  provided  the  most  useful  data. 
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OPSNION  SURVEY 


System-ID _ Group  Review  Role(s) 


DIRECTIONS:  For  each  of  the  following  items,  give  your  best  opinion  based  on  what  you  know  at  this  point. 
Respond  as  if,  in  future  reviews,  you  were  selecting  and  using  the  most  productive  group  interaction  technique. 

“Productivity”  refers  to  a  favorable  relationship  between  costs  (including  time)  and  benefits. 

•  An  “Average”  rating  indicates  approximately  equal  costs  and  benefits. 

•  “Below  Average”  indicates  that  the  costs  appear  to  be  greater  than  the  benefits. 

•  “Above  Average”  indicates  tlia^  the  benefits  appear  to  be  greater  than  the  costs. 

Survey  Rating  Scale: 

A)  Superior  B)  Above  Average  C)  Average  D)  Below  Average  E)  Poor  X)  No  Rating 


(1)  Rate  software  technical  reviewing  as  a  way  for  you  to  learn  about  software  development. 


_ (2)  Rate  software  technical  reviewing  as  a  way  for  you  to  develop  a  breadth  of  knowledge  of  application 

systems. 


_ '^3)  Rate  software  technical  reviewing  as  a  productive  way  for  an  organization  to  provide  software  quality 

assi  e. 


OPEN-ENDED  RESPONSES  (use  the  end  of  this  form  if  necessary) ; 

(4)  Say  something  positive  about  any  or  all  of  the  software  technical  reviewing  techniques  used  in  this  class. 


(  5)  Say  something  negative  about  any  or  all  of  the  software  technical  reviewing  techniques  used  in  this  class. 


(6)  Say  something,  positive  or  negative,  about  using  some  form  of  software  technical  reviewing  in  other  com¬ 
puter  science  classes. 
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(7)  Which  of  the  following  group  activities  do  you  feel  would  be  most  helpful  to  you  as  a  student  as  a  follow-up 
to  your  first  two  reviewing  activities  in  this  course?  Pick  one  (even  if  you  have  a  preference  which  isn’t  listed) 
and  comment  on  it.  If  your  preference  doesn’t  appear  on  tlie  list,  include  a  comment  about  it  below. 

(A)  Review  a  “low-level”  software,  element,  such  as  a  module  design,  or  test  plan,  or  code. 

(B)  Develop  a  “high-level”  software  element,  like  what  you  have  reviewed  so  far. 

(C)  Develop  a  “low-level”  software  element. 

(D)  Develop  a  piece  of  software  from  a  general  problem  statement  through  tested  code  (that  is,  a 
complete  team  project). 


(8)  General  CommenLs 
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Sample  Inspection  Material  with  Key  Remarks: 
System  Requirements  Definition 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Artifact  for  a  software  technical  review — a  sample  requirements  definition  and  suggestions  for  the 
results  of  the  review. 

Objectives:  To  provide  a  document  that  has  known  defects  and  is  appropriate  for  inspection. 

Prerequisites:  Students  need  to  have  an  overview  of  the  application  and  the  goals  of  the  artifact  prior  to  their 
inspection  activities. 

Procedure:  The  instructor  should  determine  the  appropriateness  of  this  document  for  the  particular  classroom 
context  The  instructor  may  want  to  correct  certain  defects  and/or  insert  additional  defects. 

Students  must  be  provided  with  a  copy  of  the  requirements  definition,  which  they  can  review  for  defects. 
Printed  copy  facilitates  the  review  process;  electronic  copy  facilitates  the  reporting  process  and  group  inter¬ 
action.  The  recommended  procedure  is  to  provide  students  with  both  forms  of  the  document. 

See  the  other  sections  of  these  support  matials  for  additional  procedures  for  administering  a  software  technical 
review. 

After  all  technical  review  output  has  been  reported,  the  instructor  should  provide  detailed  feedback,  including: 
(1)  what  defects  were  missed  and  (2)  what  remarks  were  spurious  or  wrong,  and  (3)  how  productive  the  group 
interaction  was. 

Evaluation:  The  suggested  results  of  the  review  follow  the  requirements  document.  The  instructor  should 
compare  the  students’  reviews  to  these  suggested  results. 

Each  paragraph  in  the  requirements  document  is  followed  by  a  paragraph  number  in  brackets.  The  suggested 
results  are  keyed  to  those  paragraph  numbers.  Some  of  the  suggested  remarks  are  identified  with  an  asterisk, 
denoting  a  very  important  defect,  or  with  two  asterisks,  denoting  a  critically  important  defect. 

Commentary:  Practice  materials  are  generally  hard  to  find,  so  it  is  necessary  to  take  precautions  in  order  to 
reuse  the  material  in  this  text  for  subsequent  classes.  Two  procedures  are  recommended.  First,  explain  to 
students  that  they  should  not  aid  future  users  of  this  document.  Second,  edit  the  document  with  each  use  so  tliat 
it  contains  a  different  set  of  defects. 
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lUP  Learning  Center  Tutorial  Program  Scheduling 

Project-ID: 

Document-Type: 

Document-Status: 

Author: 

Date: 

I.  Introduction 

The  lUP  Learning  Center  Tutorial  Program,  located  in  209  Pratt  Hall,  provides  tutoring  and  study  skills  coun¬ 
seling  services  to  any  lUP  student.  Currently,  mtorial  services  are  offered  in  18  disciplines  (see  SUBJECT 
listings  in  attached  pamphlet)  through  approximately  25  tutors.  Both  of  these  numbers  are  expected  to  increase 
in  the  future,  li] 

Each  tutor  is  assigned  to  one  or  more  disciplines.  That  is,  a  tutor  may  tutor  in  more  than  one  subject  area.  In  such 
cases,  one  of  the  assigned  disciplines  is  identified  as  the  “primary”  assignment  (others  as  the  “secondary” 
assignments)  for  that  tutor.  Study  skills  counseling  is  considered  to  be  a  distinct  discipline  to  which  a  few  tutors 
are  assigned.  All  the  mtoring/study  skills  counseling  takes  place  in  Pratt  Hall.  [2] 

The  “Scheduling  Desk"  carries  out  all  scheduling  and  day-to-day  administrative  activities  for  the  Tutorial  Pro¬ 
gram.  The  main  job  of  the  Scheduling  Desk  is  to  match  up  the  requests  of  students  asking  for  tutoring  (referred 
to  as  “tutees”)  with  appropriate  tutors.  Each  tutor  works  according  to  his/her  schedule,  which  is  set  at  the 
beginning  of  the  semester.  Usually,  mtoring  is  done  on  an  individual  basis,  but  some  tutors  allow  more  than  one 
tutee  to  be  present  /j/ 

'fhe  lUP  Learning  Center  would  like  to  computerize  the  scheduling  of  tutorial  appointments  to  increase  the 
efficiency  of  the  Scheduling  Desk  and  provide  better  service.  This  document  describes  the  current  system 
employed  at  the  Scheduling  Desk,  the  problems  with  the  current  system,  and  the  desired  system  to  be  developed 
in  the  future.  [4] 

n.  Current  System 

The  scheduling  system  currently  in  use  is  a  manual  one.  A  tutee  calls  or  stops  by  the  Scheduling  Desk  and 
requests  a  tutorial  session.  Tlie  tutee  will  be  asked  to  provide  the  name  of  the  discipline,  the  desired  day  and  time 
of  the  tutorial  session  (appointments  may  be  taken  up  to  two  weeks  in  advance),  and  the  tutor’s  name,  if  a 
particular  tutor  is  requested.  The  worker  at  the  Scheduling  Desk  searches  for  an  appoinmient  in  a  “Scheduling 
Book,”  a  large  binder  that  contains  the  schedules  of  all  the  tutors.  These  schedules  consist  of  sheets  of  paper, 
one  for  each  tutor  for  each  day  he/she  is  scheduled  to  work  (see  attached  sample).  On  the  sheet,  the  tutor’s  work 
hours  for  that  day  arc  indicated,  and  space  for  recording  appointments  is  provided.  These  schedules  arc  grouped 
by  the  discipline  to  which  they  are  assigned.  In  case  the  tutor  is  assigned  to  more  than  one  discipline,  his/her 
schedule  is  placed  under  the  discipline  identified  as  his/her  primary  assignment.  If  the  tutee’s  request  cannot  be 
satisfied,  the  tutee  is  asked  if  he/she  has  an  alternative  request  in  terms  of  tutor,  day,  or  time.  When  the  tutee’s 
request  is  matched  up  with  available  tutor’s  hours,  the  tutee’s  name,  phone  number,  and  name  of  the  course  (e.g., 
EC  121)  are  taken  and  written  down  on  the  scheduling  sheet  This  indicates  that  the  tutor  has  an  appointment  15) 

The  appointment  may  be  cancelled,  but  it  must  be  done  so  at  least  24  hours  before  the  appointment  time. 
Otherwise,  the  cancellation  will  be  classified  as  a  “no-show."  No-show  is  also  detected  when  a  tutee  fails  to 
keep  an  appointment.  Two  no-shows  will  disqualify  the  tutee  from  receiving  any  more  tutorial  services  for  the 
rest  of  the  semester.  The  Scheduling  Desk  is  responsible  for  keeping  track  of  the  name  of  the  tutees  who  arc 
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no-shows,  and  the  number  of  times  each  of  them  does  so.  16} 

the  day  of  the  appointment,  the  tutee  comes  to  the  Scheduling  Desk,  where  the  appointment  is  confirmed. 
Muer  the  confirmation,  the  tutee  is  asked  to  fill  in  the  “sign-in”  sheet  (a  sample  is  attached).  The  information 
recorded  on  the  sign-in  sheet  consists  of  the  tutee ’s  social  security  number,  his/her  name,  and  the  course  to  be 
tutored.  This  sheet  is  for  record  keeping  and  later  use  in  the  analysis  of  the  effectiveness  of  the  Tutorial  Program 
by  both  the  Learning  Center  administrators  and  University  administrators.  Then  the  tutor  for  that  appointment  is 
called  in  to  meet  the  tutee,  and  the  tutorial  session  begins.  Walk-in  requests  for  tutoring  are  accommodated  if 
tutors  are  available  at  that  time.  However,  no  record  of  walk-ins  appears  on  the  Scheduling  Book  since  no 
appointments  were  made  for  walk-ins.  Therefore,  the  walk-ins  are  recorded  only  on  the  sign-in  sheet.  When  the 
tutorial  session  is  over,  the  tutee  comes  to  the  Scheduling  Desk  to  “sign-out.”  This  is  done  by  the  mtee  by 
filling  in  the  column  in  the  sign-in  sheet  that  contains  the  duration  of  the  actual  tutorial  session,  counted  in 
minutes.  [7] 

A  tutor  may  change  his/her  schedule.  These  changes  are  accepted  unless  they  affect  appointments  that  have 
already  been  made.  [8] 

ni.  Problems  with  Current  System 

1.  Some  information  is  taken  repeatedly.  The  information  that  is  collected  at  the  time  of  making  an  appointment 
overlaps  with  the  information  taken  at  the  sign-in  time.  This  is  not  only  inefficient,  but  it  also  causes  congestion 
around  the  Scheduling  Desk  by  tutees  waiting  in  line  to  fiU  in  the  sign-in  sheet.  [9] 

2.  Procedures  for  making  an  appointment  for  tutors  who  are  assigned  to  more  than  one  discioline  are  not 
efficient  when  the  appointment  is  not  for  the  primary  assignment.  The  worker  at  the  Scheduling  Desk  mu.st  flip 

lages  of  tlie  Scheduling  Book  to  get  to  the  section  where  the  tutor’s  schedule  is  placed.  This  situation  occurs 
s.,  jjrisingly  often,  especially  when  all  tutors  who  are  assigned  to  the  discipline  are  filled  up  to  capacity,  and 
tutors  who  have  a  secondary  assignment  to  that  discipline  must  be  checked.  [lO] 

3.  Procedures  for  confirming  the  appointment  when  uie  tutee  comes  in  are  inefficient.  This  is  especially  due  to 
the  fact  that  all  the  appointments  start  on  the  hour  or  half  hour,  resulting  in  most  of  the  tutees  coming  in  at  the 
same  time.  Qass  schedules  also  influence  these  peak  periods  of  activity  at  the  Scheduling  Desk,  [iij 

4.  Checking  for  no  shows  is  neither  efficient  nor  reliable.  This  is  because  the  worker  at  the  Scheduling  Desk 
must  cross  check  the  times  and  names  in  the  Scheduling  Book  against  the  times  and  names  on  the  sign-up  sheet 
in  order  to  determine  who  failed  to  show  up  for  an  appointment.  112] 

5.  The  record  on  the  sign-in  sheet  is  organized  in  order  of  arrival  lime,  but  an  alphabetical  ordering  would  be 
easier  to  refer  to.  Also,  use  of  the  sign-in  sheet  results  in  the  loss  of  confidentiality  concerning  the  information 
on  the  tutees,  since  all  the  tutees  use  the  same  sign-in  sheet  where  they  can  easily  see  the  information  on  the 
tutees  who  came  in  before  they  did.  113} 

6.  Currently  appointment  restrictions  set  by  tutors  are  not  observed  by  personnel  at  the  Scheduling  Desk.  For 
example,  a  math  tutor,  for  one  reason  or  another,  may  not  feel  comfortable  helping  someone  taking  Calculus  II, 
and  thus  instruct  the  Scheduling  Desk  not  to  lake  appointments  for  that  course.  Another  example  is  that  some 
tutors  who  do  accept  group  appointments  may  require  that  all  the  tutees  who  would  be  present  in  one  session  to 
be  from  the  sections  taught  by  the  same  professor.  These  appointment  restrictions  are  not  observed  well.  ii4] 

"Tiere  are  no  procedures  for  recording  which  worker  at  the  Scheduling  Desk  took  which  appointment  This 
t  -cd  some  inconvenience  in  the  past  when  disputes  arose  about  the  time/day  of  an  appointment,  or  the  exis¬ 
tence  of  the  appointment  itself.  [15] 
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8.  The  Scheduling  Book  is  not  updated  on  a  daily  basis.  The  schedules  that  contain  the  appointments  for  the 
day  should  be  removed  at  the  end  of  the  day  and  new  schedules  that  contain  appointments  for  two  weeks  from 
the  current  day  should  be  inserted.  This  is  done  only  once  a  week  simply  because  the  procedure  takes  some 
time,  and  the  Scheduling  Book  cannot  be  kept  away  from  the  Scheduling  Desk  for  that  time  everyday.  Workers 
at  the  Scheduling  Desk  also  dislike  the  “flip- flopping"  of  the  pages  of  the  bulky  Scheduling  Book.  This 
problem  may  be  reduced  somewhat  by  devising  a  new  more  efficient  manual  system  instead  of  computerizing 
the  Scheduling  Desk  operations.  116] 

IV.  System  Goals  and  Requirements 

As  mentioned  in  the  last  section,  it  may  be  possible  to  improve  the  efficiency  and  effectiveness  of  the  Schedul¬ 
ing  Desk  by  modifying  the  current  manual  system.  However,  the  whole  operation  of  the  Learning  Center,  of 
which  the  Tutorial  Program  is  a  part,  is  expected  to  go  through  the  transformation  from  a  collection  of  manual 
systems  to  an  integrated  computerized  system.  It  is  desired  that  operation  of  the  Tutorial  Program  be  comput¬ 
erized,  not  only  for  its  sake,  but  also  as  a  part  of  a  larger,  integrated  system.  [I7j 

Though  the  Scheduling  Desk  is  only  a  part  of  the  Tutorial  Program,  it  is  where  most  of  the  data  concerning  the 
Tutorial  Program  are  collected,  and  thus  the  computerization  of  the  Scheduling  Desk  will  be  a  major  part  of  the 
computerization  of  the  operations  of  the  Learning  Center.  The  operations  of  the  Scheduling  Desk  will  be  the 
first  major  manual  system  to  be  computerized,  and  therefore,  the  system  to  be  developed  for  the  Scheduling 
Desk  must  be  modifiable  so  that  it  can  acconunodate  future  changes  in  the  requirements.  Future  changes  to  the 
system  are  expected  to  concern  the  addition  of  new  functions  to  the  system,  and  the  fonn  and  content  of  the 
output  file  created  by  the  system  for  use  by  other  Leamirg  Center  operations.  Current  goals  and  requirements 
for  the  system  to  be  developed,  togetlier  with  a  rationale  for  their  inclusion,  are  presented  below,  (isj 

1.  The  system  must  be  highly  user-friendly.  —  The  people  using  the  system  cannot  be  expected  to  have  any 
computer  experience  and  the  turnover  rate  of  the  personnel  will  be  relatively  high.  //?/ 

2.  The  system  must  be  highly  robust  —  For  the  same  reason  as  specified  in  lequirement  1,  the  system  should 
respond  to  user  errors  by  detecting  them,  diagnosing  the  cause,  and  offering  helpful  suggestions  about  how  to 
correct  them.  [20] 

3.  The  response  time  of  the  system  must  be  reasonably  fast,  less  than  five  (5)  seconds  is  desired.  —  Quick 
response  is  crucial  since  the  demands  placed  on  the  Scheduling  Desk  tend  to  be  concentrated  on  certain  times, 
namely  a  few  minutes  before  and  after  the  hour  or  the  half  hour.  [2i] 

4.  The  system  must  be  interactive  and  flexible  enough  to  accommodate  all  the  possible  requests  from  tutees.  — 
The  most  often  heard  requests  from  tutees  arc  for;  an  appointment  anytime  during  certain  hours,  an  appointment 
with  any  tutor  available,  and/or  a  list  of  all  the  time  slots  during  which  a  particular  tutor  will  be  available.  [221 

5.  Appointments  should  be  scheduled  for  either  1/4,  1/2,  or  one  hour,  as  requested  by  the  tutee.  The  system 
must  be  able  to  accommodate  any  combination  of  such  appointments.  [23] 

6.  The  system  should  be  developed  in  such  a  way  that  all  the  necessary  information  on  tutees  and  the  courses  in 
which  they  wish  to  be  tutored  are  taken  when  the  appointments  are  taken.  —  This  would  ease  the  confirmation 
of  the  appointment  at  the  sign-in  time  since  no  new  information  must  be  taken  at  that  time.  [24] 

1.  The  system  must  provide  for  easy  confimiation  of  an  appointment  at  the  sign-in  time.  No  duplicate  entry  of 
the  data  is  desired.  However,  corrections  to  the  information  on  the  tutees  and/or  the  course  to  be  tutored  may  be 
made  at  this  time.  —  The  Scheduling  Desk  worker  who  takes  the  appointment  might  make  mistakes  in  record¬ 
ing  data.  Also,  a  number  of  tutees  do  not  know  the  exact  name  of  the  course  in  which  they  wish  to  receive 
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tutoring,  thus  giving  the  Scheduling  Desk  workers  incorrect  information.  /25/ 

8.  TTie  system  must  be  able  to  detect  no-shows  automatically,  and  keep  track  of  all  the  no-shows  throughout  the 
semester.  —  When  a  tutee  accumulates  two  no-shows,  the  system  must  cither  be  set  to  automatically  reject 
requests  from  tliat  tutee  in  the  future  or  print  out  the  name  of  the  tutee  so  that  the  Scheduling  Desk  workers  will 
be  informed  of  the  disqualification  of  the  tutee.  126! 

9.  The  system  must  be  able  to  handle  cancellations.  — Regardless  of  whether  the  cancellation  is  made  within  24 
hours  of  the  appointment  or  not,  the  cancellation  should  make  the  tutor  f.vailable  for  new  appointments  for  the 
time  of  the  canceled  appointment.  If  the  cancellation  is  made  within  less  than  24  hours  of  the  appointment,  it 
should  be  classified  as  a  no-show.  [27] 

10.  The  system  must  provide  for  usable  back-up  in  case  of  a  system  failure.  —  Regardless  of  the  hardware 
used,  it  is  essential  to  have  ways  to  schedule  and  meet  appointments  in  case  the  system  fails.  [28j 

1 1.  Special  restrictions  placed  by  individual  tutors  must  be  displayed  on  the  screen  when  the  appointments  are 
taken  for  those  tutors.  —  The  Scheduling  Desk  worker  must  see  the  restrictions  before  he/she  takes  an  appoint¬ 
ment  for  that  tutor.  However,  the  restrictions  are  to  be  displayed  for  the  Scheduling  Desk  workers’  information 
only,  and  the  system  should  accept  appointments  that  are  not  in  accordance  with  the  restrictions  placed  by  the 
tutor.  In  other  words,  the  Scheduling  Desk  workers  must  be  aware  of  tutor  restrictions,  but  they  must  also  be 
able  to  override  them.  [29] 

12.  The  system  must  be  able  to  accommodate  group  appointments.  —  The  system  must  allow  more  than  one 
tutee  to  be  scheduled  for  one  appointment  Note  that  the  two  (or  more)  tutees  may  contact  the  Scheduling  Desk 
at  different  times.  Thus,  the  system  must  allow  for  scheduling  another  tutee  at  the  same  time  slot  as  an  already 
taken  appointment  [30] 

13.  The  system  must  be  able  to  handle  tutors  who  have  multiple  assignments.  —  The  system  is  expected  to  free 
the  Scheduling  Desk  workers  from  the  job  of  memorizing  which  tutors  have  multiple  assignments  and  which 
disciplines  are  their  primary  disciplines.  For  example,  if  a  tutor  who  is  assigned  to  French  and  Mathematics  has 
an  hour  available  for  tutoring  on  Tuesday,  that  hour  should  appear  as  vacant  regardless  of  whether  the  appoint¬ 
ment  is  sought  in  French  or  Mathematics.  [3t] 

14.  The  system  must  be  able  to  make  an  appointment  up  to  two  weeks  in  advance  at  any  time.  —  The 
computerized  Scheduling  Book  must  maintain  and  use  complete,  current  scheduling  data.  [32] 

1.5.  The  system  must  be  able  to  accommodate  both  temporary  and  permanent  changes  in  the  schedules  of  tutors. 
—  If  the  notification  of  changes  is  received  more  than  two  weeks  in  advance,  the  changes  must  be  accepted  and 
schedules  must  be  modified  to  reflect  those  changes.  If  changes  within  the  next  two  weeks  are  requested,  the 
system  must  check  for  appointments  already  made  that  would  be  affected  by  the  changes.  If  there  are  any  such 
appointments,  tlie  request  for  schedule  changes  will  be  denied.  Otherwise,  the  request  will  be  accepted.  [33] 

16.  The  system  must  be  able  to  display  all  the  appointments  a  tutor  has  for  the  day.  —  The  tutors  very  often  ask 
the  Scheduling  Desk  to  show  them  all  their  appointments  for  the  day  when  they  come  in  for  work.  [34] 

17.  The  system  must  be  able  to  “block”  appointments  for  all  the  tutors  for  certain  hours  and/or  days.  —  There 
are  times  when  no  tutorial  services  are  offered,  such  as  holidays,  the  eveni  3  before  the  break,  or  the  time  when 
all  the  tutors  are  required  to  attend  a  Learning  Center  meeting.  [35] 

18.  The  system  must  produce  a  .summary  of  the  activities  of  the  day  at  the  end  of  the  day  or  at  the  beginning  of 
the  next  working  day.  —  A  file  that  contains  information  on  tutorial  sessions  that  took  place  during  the  day. 
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sorted  alphabetically  on  the  tutees’  last  names,  must  be  created.  Also,  tlie  system  must  be  able  to  provide  a  daily 
report  of  the  number  of  tutorial  sessions  that  take  place,  grouped  by  the  disciplines.  [36i 

19.  The  system  is  expected  to  offer  security  for  the  hardware  as  well  as  the  software,  including  the  data  files.  — 
The  record  of  tutorial  sessions  as  well  as  the  record  of  appointments  arc  confidential  and  should  not  be  acces¬ 
sible  to  anyone  who  does  not  work  at  the  vSeheduling  Desk,  except  tne  Learning  Center  administrators.  Also, 
certain  functions  of  the  system,  especially  the  ones  that  summarize  the  activities  of  the  day,  may  not  be  acces¬ 
sible  to  anyone  except  Learning  Center  administrators.  I37j 

20.  The  system  must  record  the  duration  of  the  tutorial  session  (measured  in  minutes)  after  the  session  is  over. 
—  Currently,  the  information  concerning  the  number  of  minutes  of  the  actual  tutorial  session  must  be  input  to 
the  system  retrospectively  and  somehow  associated  with  the  recotd  of  that  tutorial  session.  [38] 

71.  The  system  must  request  the  initials  (or  some  other  forms  of  identification)  of  the  Scheduling  Desk  workers 
at  each  step  of  the  operation.  —  It  is  desired  that  the  system  will  not  let  the  user  proceed  unless  the  user 
identifies  him/herself.  This  is  desired  in  order  to  keep  Scheduling  Desk  workers  highly  disciplined  as  well  as  to 
offer  some  measure  of  security.  (39i 

22.  When  the  appointment  is  made  in  person  (i.e.,  the  tutee  comes  to  the  Scheduling  Desk  to  make  an 
appointment),  the  system  should  be  able  to  print  locally  a  memo  that  shows  the  day  and  the  time  of  the  appoint¬ 
ment,  and  the  names  of  the  tutor  and  tutee.  —  This  is  desirable,  but  it  is  not  necessary.  [40] 

V.  System  Constraints 

The  system  may  be  implemented  on  either  the  locally  available  mainframe  or  the  microcomputers  (IBM  PCs) 
owned  by  the  I-eaming  Center.  If  microcomputers  are  to  be  used,  the  system  will  communicate  to  other  comput¬ 
erized  Learning  Center  operations  by  transferring  its  files  to  the  mainframe  through  the  use  of  a  modem  and  a 
phone  line.  [41  j 
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De  Remarks 

1.  Ill  **  How  much  increase,  and  in  what  time  frame? 

2.  Ml  This  document  must  "define  requirements  for  a  new  system,"  rather  than  "define  a  new 
system." 

3.  isj  What  if  there  is  no  match? 

4. 16}  *  What  data  are  recorded?  How  are  files  maintained?  What  reports  or  lists  arc  needed?  What 
volume  of  data  is  involved?  What  use  is  made  of  these  data? 

5. 19J**  Specifically  which  data  are  redundant?  How  do  inconsistencies  affect  the  overall  system? 

6. 1141  Where  are  these  restrictions  recorded?  What  arc  these  restrictions?  Is  some  improvement  in 
handling  these  restrictions  a  desired  feature  of  the  new  system?  How  can  a  restriction  be  specified 
for  a  particular  instmetor’s  students,  when  there  is  no  specification  for  how  that  restriction  is 
recorded  in  the  system? 

7. 116J  *  The  current  operational  detail  is  not  clear.  Tliis  document  gives  impression  that  appoint¬ 
ments  are  entered  as  they  are  received.  Also,  what  is  done  with  old  schedules  when  they  are 
removed? 

8. 1/6/  The  goal  of  this  document  is  to  state  requirements  for  the  new  system,  not  patches  for  the  old. 

9. 117]  *  Who  desires  the  computerized  system?  (The  use  of  passive  voice  is  a  sign  that  something  is 
not  stated.) 

10.  //tf/ The  total  context  is  not  yet  defined,  so  integration  criteria  cannot  be  stated.  Without  an 
jventU  plan  into  which  the  proposed  system  it,  the  TUTOR  system  is  premature. 

1 1 , 119}  The  term  "user-friendly"  requires  precise  definition. 

12  120]  The  term  "robust"  also  requires  precise  definition. 

13. 120]  Requirements  1  and  2  overlap  significantly.  Combine  them? 

14. 121]  *  Response  to  what? 

15. 122]  *  Only  tho.se  requests  that  arc  documented  at  this  point  can  be  expected  to  be  dealt  with  in  the 
new  system.  Any  additional  types  of  requests  may  have  to  be  built  into  the  system  as  an 
(expensive)  CHANGE. 

16. 124]  *  What  is  “ail  the  necessary  info”? 

17. 125]  Requirements  6  and  7  are  redundant. 

18. 126]  *  What  specific  system  response  is  desired;  beep  and  print  a  message,  produce  periodic  re¬ 
ports,  flag  a  student  against  future  appointments,  cancel  all  existing  appointments, ...  ? 

19. 130]  *  How  many  tutees  max?  Are  there  any  room  restrictions?  Are  tutees  allowed  to  have  special 
restrictions  on  group  tutoring,  or  are  there  special  restrictions  by  the  topic  area  or  tutee  needs? 

20,  [31]  *  How  does  a  ume  “appear  as  vacant”?  How  many  areas  of  expertise  may  tutor  have?  What 
are  the  current  areas  of  expertise,  and  how  are  they  coded?  What  growth  in  this  area  is  an¬ 
ticipated?  What  are  the  total  effects  of  primary  and  secondary  assignments,  and  is  there  any  an¬ 
ticipated  growth  in  this  area? 

7/  The  meaning  of  “complete”  and  “current”  should  be  stated.  Also,  note  that  “any  time”  could 
be  construed  to  mean  any  time  of  day  or  night. 
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22.  /J5/Thc  tenn  “block”  should  be  precisely  defined. 

23. 1361  *  This  paragraph  is  "design  constraining."  Note:  Arc  the  reported  data  to  be  limited  to  tutee 
name,  discipline,  and  time? 

24.  [371  *  Exactly  what  are  the  security  concerns:  theft,  vandalism,  prying,  hacking,  unauthorized  per¬ 
sonal  use  of  equipment, ...  ? 

25. 1391  *  The  requirement  to  force  system  users  to  initial  things  is  design  constraining,  and  it  may  not 
be  a  good  solution  to  the  need  for  security  and  discipline.  Also,  items  19  and  21  overlap. 

26.  HI/  Alternatives  to  file  traiiifcrs  are  possible.  Is  there  a  particular  reason  for  insisting  on  file 
transfers? 

27.  !■#//  **  Constraints  on  budget,  development  time,  and  development  staff  are  not  stated. 

28. 141]  *  What  are  the  estimated  data  volumes,  processing  requirements,  and  growth  predictions? 

Summary  and  Evaluation  Report 

(1)  This  document  fails  to  define  the  context  in  which  SOLAR  will  be  required  to  function.  System  devel¬ 
opment  should  not  proceed  until  this  issue  is  resolved. 

(2)  User  expectations  for  this  system  are  unrealistic  and  incompatible.  This  problem  must  be  considered  when  a 
high-level  design  is  developed  for  an  actual  SOL/vR  system. 

Status  of  software:  NOK. 
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Directions  for  Software  Technical  Reviews  by  Students 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Instructor  procedures  for  administration  of  software  technical  reviews;  handouts  for  students. 

Prerequisites:  In  order  to  be  effective  at  a  group  process  for  detecting  defects,  reviewers  must  have  the  follow¬ 
ing  types  of  knowledge  and/or  skill.  Student  learning  or  skill  development  in  the  following  areas  may  be  more 
of  a  goal  than  a  prerequisite,  but  the  instructor  should  consider  all  of  the  following  concerns  in  his  or  her 
planning.  The  concerns  are: 

•  Knowledge  of  software  application  domain 

•  Technical  knowledge  of  the  type  of  software  and  its  role  in  a  software  development  life  cycle 

•  Active  strategies  for  detecting  defects 
o  Writing  skill  to  document  defects 

•  Oral  communication  skill  to  participate  in  group  meetings 

•  Knowledge  of  procedures  and  policy  for  a  technical  review  project 

•  Knowledge  of  different  points  of  view  of  participants  in  software  technical  reviews. 

Procedure:  Procedures  are  described  on  the  next  page.  This,  along  with  the  various  student  handouts  following 
should  be  read.  Instructors  can  then  create  their  handouts  as  required. 

Evaluation:  'Phe  procedures  used  for  technical  software  reviews  must  be  evaluated  by  management — in  this 
case,  the  instructor.  The  following  data  can  be  used; 

•  Detailed  grading  data  for  both  individual  and  group  performance 

•  Historical  data  from  previous  reviews  (and  reviews  in  other  contexts) 

•  A  Student  Opinion  Survey  (p.  36) 

Commentary;  The  strengths  and  weaknesses  of  alternative  procedures  arc  listed  and  discussed  in  Deutsch  and 
Willis  [Deutsch88]  and  the  “Notes  on  Software  Technical  Reviewing”  (p.  4).  Note  that  the  software  to  be 
reviewed  is  an  important  element  in  making  a  software  technical  review  effective.  The  fundamental  concerns 
can  be  summarized  as  follows: 

•  Inspectable  software 

•  Compatible  knowledge  of  reviewer 

•  Complementary  knowledge  of  group 

•  Active  strategies  for  detecting  defects 
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Procedures  for  Technical  Software  Reviews 

h  .111(1  Willis  (Oi»i)lscht\tt|  li.si  livt?  (worall  I'aciors  in  ucliicviin*  elTcctivc  quality  enhancement  through 
(diii.il  s  'iiw.iir  u'chiiual  ivview,  I'lu*  Him  ilux'o  of  ihc.se  lactors  apply  directly  to  all  student  technical  reviews, 
hv'x*  lii('i(U>.  aie 

•  .'ll  ovcivicw  ot  ihc  technical  ivviewing  tiisk 

•  Individual  pix'p.iraiion  lor  grx'up  ititeniciion  (allow  4-7  days) 

•  I  iioiip  interaction  tallow  up  to  t  days  to  documetit  gtxnip  commenLs) 

•  Kewotk 

•  l  ollow  up 

he  iiistnictoi  must  distnbute  handouts  to  e.xplain  and  document  ba,sic  procedures  and  guidelines.  Thc.se  hand- 
.it.s  must  sivcilv  pixvedutx's  lor  studotus,  cx|x'ctaiion.s  for  individual  reviewing  output,  and  expectations  for 
rou|'  .ictiviiies  Haste  materials  am  provided  in  this  section,  but  tlicrc  arc  significant  alternatives  that  must  be 
uisideivil  relative  to  the  learning  goals  and  work  context  for  a  particular  .software  technical  reviewing  activity, 
ih'  maienals  included  in  this  section  require  the;  instructor  to  select  one  handout  for  individual  reviewing 
ivciioiis  and  an  additional  handout  for  group  activities.  'Phese  materials  do  not  exhaust  the  viable  options,  but 
e\  (l(»  pmvidc  adequate  direction  to  implement  a  valuable  learning  activity  in  most  university  learning  situa- 
nis 

le  suggested  pixH'edurv  for  refxirting  comments  by  individuals  and  groups  is  to  have  students  key  in  their 
m  '  s  If  the  envitxuuncnt  allows  it.  'Hie  implementation  of  machine-readable  remarks  facilitates  the  pre- 
1  ..iiioii  pnvedure,  the  group  meeting,  and  the  analysis  of  everytlting  that  happens  in  the  review.  This 

ipixwed  capability  enhances  die  potential  learning  bencfiLs  that  can  be  gained  from  the  activity  by  facilitating 
.inah,  SIS  ol  its  outcomes,  ;uid  it  decreases  the  paperwork.  However,  a  manual  procedure  for  recording 
lividual  ix'iuaiks  ciut  lie  a  viable,  organized  approach  to  software  technical  reviews;  the  absence  of  a  suitable 
St  coiiipuiing  coniexi  should  not  affect  the  decision  of  whether  or  not  to  implement  a  software  technical 
■'icw  Kig  ,;ciiviiy  in  a  particular  Icaining  context. 

Icci  ,1  l.icc  !('  face  method  of  group  interaction  when  students  arc  uncertain  of  how  to  conduct  themselves  in  a 
.'icw  riic  computer-mediated  group  interaction  procedure  is  most  appropriate  when  an  analysis  of  the  group 
n.imus  (It  the  ix’vicwing  activity  are  a  key  concern  for  student  learning  or  instructor  research.  Cornputcr- 
'di.ilod  group  interaction  al.so  allows  a  geographically  dispersed  group  of  students  to  work  together.  Indi- 
luai  ivs[xm.so  to  the  merged  remarks  ol  a  group  of  reviewers  is  a  convenient  procedure  because  it  takes  less 
le  and  it  divs  not  roquiiv  studen's  to  cooperate  with  each  odicr.  'Phis  individualized  proc^'^dure  provides 
nificant  learning  benefits  with  minimal  cost,  but  procedures  that  involve  group  interaction  are  preferable 
cause  of  the  imjxrrtancc  ol  group  effort  to  professional  software  development. 

e  choices  tor  a  sp<'cific  technique  for  group  interaction  must  be  carefully  considered — no  single  technique 
isllcs  all  the  techniciil  review  needs  of  current  best  practice.  The  Fagan  (Fagan76]  “Inspection”  technique  is 
most  powerful  procedure  for  evoking  group  synergism.  However,  it  can  fail  if  students  do  not  understand 
roic.s  of  client,  autiior,  consumer,  and  tester.  The  Freedman  and  Weinberg  [Freedman82]  technical  review 
cedure  is  less  dcmmiding  (and  less  powerful).  Both  techniques  require  leadership  from  the  review  leader  and 
ro'igh  records  of  the  group’s  findings. 

;i>iiituonded  group  sizes  are;  3  or  1  for  computer-mediated  group  interaction,  4  or  5  for  a  review  meeting 
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[Freedman82],  and  4  to  6  for  an  inspection  [Fagan76].  In  every  case,  the  entire  project  should  take  no  less  than 
5  days  and  no  more  than  14  days. 

All  technical  review  activities  in  an  academic  setting  include  the  following  elements.  (See  also  the  “Instructor’s 
Checklist,”  p.  22.) 

•  Selection  and  dissemination  of  procedures  for  students 

•  Overview  of  software 

•  Individual  software  technical  review 

•  Group  interaction 

•  Instructor  and/or  smdent  evaluation  of  student  performance 

•  Feedback  to  students 

•  Instructor  evaluation  of  each  project 

•  Instructor  recording  of  data  on  outcome  of  the  activity 

•  Instructor  monitoring  of  historical  data 
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Student  Handouts 


On  the  next  several  pages  are  handouts  or  sections  of  handouts  suitable  for  distribution  to  students.  The  instruc¬ 
tor  should  choose  from  these  as  needed,  depending  upon  the  type  of  reviews  being  conducted  and  the  adminis¬ 
trative  procedures  appropriate  to  the  educational  environment. 


General  Reviewing  Guidelines 

The  attached  list  of  categories  summarizes  a  lot  of  accumulated  experience,  so  you  may  find  it  helpful  to  use  it 
as  an  initial  set  of  clues  about  what  to  look  for.  Your  understanding  of  what  must  be  stated  in  software  devel¬ 
opment  documents  and  the  intended  audience  for  various  documents  must  be  your  basic  guideline  for  what 
might  constitute  helpful  things  to  say  in  a  review.  Your  eventual  goal  should  be  to  detect  things  worthy  of  a 
comment,  write  your  comment,  and  then  consult  the  category  list  to  determine  a  classification  for  your  comment. 

You  will  be  tempted  to  comment  on  the  style  of  prose  or  code  in  the  software.  Comment  on  style  only  when  the 
underlying  meaning  is  wrong,  ambiguous,  extraneous,  or  incomplete,  If  a  style  standard  for  your  particular 
software  environment  is  clearly  violated,  document  the  situation  as  a  defect.  Other  concerns  about  the  style  of 
the  software  reviewed  may  simply  be  listed  as  separate  “Related  Issues”  (p.  64),  but  they  are  not  to  be  reported 
as  defects  in  the  software.  With  regard  to  reviewing  prose  errors  in  spelling  or  grammar,  they  may  be  notes,  but 
doing  so  will  not  contribute  to  the  performance  score  for  individuals  or  groups  in  this  software  technical  review. 

Classification  of  reviewer  comments  facilitates  the  analysis  of  review  output,  and  it  also  seems  to  produce  more 
thoughtful  reviews.  The  “General  Remark”  category  can  be  very  important,  since  any  list  might  blind  the 
reviewers  to  issues  not  anticipated  by  the  writer  of  the  list  The  primary  concern  in  noting  points  about  the 
software,  being  reviewed  should  be  whether  or  not  your  comment  helps  the  software  document  achieve  its  goals 
as  well  as  possible  for  as  long  as  it  is  needed. 

You  are  expected  to  work  independently  and  discuss  this  project  only  at  the  formally  arranged  times.  Please 
report  all  activities  that  may  be  relevant  to  your  performance  on  the  reviewing  tasks,  so  that  your  instructor  can 
make  correct  inferences  about  the  source  of  your  fantastic  performance.  A  time  reporting  form  is  attached  [not 
included  in  these  support  materials)  for  reporting  all  time  that  you  spend  on  this  project.  Please  fill  it  out 
faithfully  every  time  you  do  something  relevant  to  this  project. 


Individual  Preparation  for  a  Software  Technical  Review 

(0)  Participate  actively  in  an  overview  meeting. 

(1)  Study  your  handouts  and  notes  until  you  have  a  clear  idea  of  the  goals  of  the  document  that  you  are  review¬ 
ing.  Review  the  types  of  defects  that  have  been  detected  in  past  experience  with  similar  pieces  of  software. 

(2)  Read  a  printed  copy  of  the  document  tliat  you  are  to  review.  Get  a  general  feel  for  what  it  does  and  how  it  is 
organized.  You  will  probably  work  more  efficiently  if  you  wait  until  your  second  reading  before  you  make  any 
extensive  notes  about  defects,  since  some  of  your  concerns  will  be  resolved  when  you  have  read  the  whole  thing. 

(3)  Reread  the  document  and  mark  the  points  that  you  feel  are  worthy  of  a  reviewer  comment.  Yot;  may  want  to 
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mark  these  points  with  a  circled  number  for  ease  of  reference.  Write  your  comments  on  a  separate  sheet  of 
paper.  Make  an  electronic  copy  of  the  software  that  you  are  reviewing.  Edit  your  comments  into  tlie  document 
under  the  line  that  best  points  to  each  specific  concern.  Each  comment  must  include  a  category  from  tlie 
attached  list  of  categories  for  remarks  about  requirements  [p.  66].  (Note  that  you  may  elect  to  record  your 
comments  directly  into  electronic  form,  thus  skipping  the  transcription  process.  Do  whatever  seems  most  com¬ 
fortable  to  you,  but  picxJuce  a  document  that  has  the  same  form  as  the  sample.) 

(4)  Reread  t'  r  document  and  your  remarks.  Qieck  the  software  that  you  are  reviewing  one  more  time  for  such 
genentl  thing's  as  verifial  ility,  .specificity,  feasibility,  and  completeness.  Compare  the  overall  structur :  to  your 
class  notes  'orid  the  particular  guidelines  that  you  will  be  given  for  each  type  of  document  that  you  review. 
Revise,  your  comments  as  neeried. 

(5)  Submit  your  edited  copy  of  the  reviewed  document  according  to  the  guidelines  in  the  "Document  Submis- 
sii.ui  Procedure”  at  the  end  of  these  notes.  At  the  time  when  individual  review  comments  are  due,  any  student 
who  is  not  eady  will  be  excluded  from  the  group  review.  This  will  hurt  both  the  individual  and  the  group,  so  do 
not  miss  deadlines! 


Group  Reviewing  Procedure 

Synergism  is  a  key  concept:  the  gi\'/up  review  technique  should  yield  results  that  are  noticeably  better  than  the 
sum  of  individual  reviewer  comments.  This  means  that  the  group  review  should  combine,  filter,  extend,  clarify, 
and  summarize  the  individual  comments.  In  addition,  new  insight  may  be  triggered  by  the  remarks  of  other 
group  members. 

Your  specific  procedure  is  given  on  a  separate  handout,  in  order  to  allow  you  instructor  the  flexibility  to  choose 
the  most  appropriate  procedure  in  your  context.  The  required  group  documeni  is  a  final  form  of  the  review 
comments  (cf.  “Sample  Software  Defect”  [p.  12]),  and  a  summary  evaluation  report  of  the  software  that  was 
reviewed  (cf.  “Inspection  Summary  and  Evaluation  Report  Form”  [p.  61]). 


Grading  Policy  Grading  Policy  f  jr  ftware  Technical  Reviewing 

In  f lattice,  software  reviews  should  never  be  used  for  performance  evaluation,  since  this  could  harm  their 
cffec:ivcness  a.s  a  technique  for  achieving  software  quality  goals.  Managers  should  try  to  monitor  the  effec¬ 
tiveness  of  reviews,  but  tliey  should  never  use  them  for  performance  evaluation.  However,  in  software  engi¬ 
neering,  stutients  will  be  evaluated  on  their  performance  on  software  reviewing  tasks,  since  learning  is  the  goal 
of  the  course.  A  composite  grade  will  be  assigned  for  each  task  as  follows: 

Individual  comments  15  points 
Group  output  15  points 

(each  member  of  the  group  will  receive  the  same  score) 

Time  reporting  2  points 

Total  32  points 

(Coordinator/recorder  2  bonus  points) 

Note  two  important  things  about  grades.  First,  submission  of  your  documents  must  be  on  time,  as  determined  by 
their  deadlines  and  the  following  “Document  Submission  Procedure.”  If  the  average  group  output  scores  for  a 
particular  reviewing  technique  appear  to  be  significantly  different  from  the  other  groups  because  of  a  particular 
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reviewing  technique,  your  instructor.  Your  instructor  may  adjust  the  lower  group  scores  upward  in  order  to 
compensate  for  any  unfairness  in  grading  that  might  otherwise  occur. 


Document  Submission  Procedure 

Mail  your  annotations  to  your  instructor  (alternatively,  “your  group  review  leader”).  A  single  copy  of  the 
software  witli  all  of  your  group’s  annotations  merged  into  it  will  be  made  available  to  you  at  your  group  meeting 
(alternatively,  “at  some  specified  time  prior  to  the  group  meeting”). 


References 

Copies  of  all  references  are  available  under  your  instructor’s  name  at  the  library  reserve  desk. 
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Face-to-Face  Technical  Review  Meeting 


At  the  time  when  individual  comments  are  handed  in,  your  group  must  choose  a  coordinator  and  two  recorders. 
Responsibilities  for  each  of  these  roles  are  detailed  below.  Under  the  leadership  of  this  newly  selected  coor¬ 
dinator,  the  group  then  chooses  a  time  for  a  two-hour  group  meeting.  Schedule  this  meeting  within  four  days, 
since  your  preparation  for  the  meeting  must  be  fresh  in  your  minds.  Allow  time  after  the  group  meeting  for  your 
recorder  to  prepare  your  final  document  before  the  due  date. 

Everyone  in  the  group  must  participate  in  the  entire  group  meeting  in  order  to  receive  group  reviewing  points. 
Your  instmctor,  or  an  official  representative,  will  observe  the  meeting  and  make  an  audio  recording  of  what  is 
said.  The  meeting  should  take  no  longer  than  two  hours. 

Coordinator  Responsibilities:  The  coordinator’s  first  responsibility  is  seeing  that  the  arrangements  for  the 
meeting  are  all  taken  care  of.  This  includes  seeing  that  everyone  gets  there  on  time.  In  a  professional  setting, 
the  coordinator  would  also  schedule  a  room,  prepare  copies  of  necessary  materials,  and  see  that  all  the  review 
participants  are  prepared  for  the  meeting.  In  this  case,  your  instructor  will  take  care  of  the  room  and  the  review 
materials.  Any  group  member  who  fails  to  prepare  for  the  meeting  will  already  have  been  eliminated  at  the 
individual  review  submission  time. 

The  coordinator’s  second  responsibility  is  to  act  as  a  meeting  leader.  At  the  group  review  meeting,  the  coor¬ 
dinator  should  obtain  the  group’s  agreement  on  what  must  be  accomplished  in  the  meeting,  and  then  see  that  the 
meeting  goals  are  achieved  as  well  as  possible.  This  means  that  the  coordinator  should  keep  the  discussion  on 
track  and  keep  an  eye  on  the  time.  The  coordinator  must  ensure  that  the  individual  comments  are  understood, 
merged,  edited,  or  deleted,  in  whatever  manner  the  group  feels  might  be  helpful.  The  old  comments  should  be 
extended,  and  new  comments  generated  whenever  possible.  Your  group’s  final  reports  on  the  software  that  is 
being  reviewed  must  state  overall  strengths  and  weaknesses  and  emphasize  major  issues,  so  be  especially  alert 
for  main  themes  throughout  the  two  hour  period.  If  agreement  cannot  be  reached  on  any  point,  consider  the 
most  negative  opinion  to  be  the  review  outcome,  but  report  that  there  was  disagreement  on  that  point.  The 
coordinator  is  also  responsible  for  deciding  whether  or  not  to  take  a  break. 

The  coordinator  slKJuld  not  have  to  use  heavy-handed  or  autocratic  tactics,  since  the  reviewers  are  competent, 
well-prepared  (almost)  professionals.  The  tools  of  the  coordinator  include  a  variety  of  group  interaction  tactics. 
The  document  can  be  reviewed  in  some  structured  fashion  (top-down  is  generally  better  than  beginning  to  end), 
everyone  can  be  asked  to  say  something  good  and  bad  about  the  product,  one  reviewer  can  be  asked  to  blast  it 
and  another  to  praise  it,  or  walk-throughs  or  other  pseudo-simulations  can  be  used.  In  this  situation,  we  recom¬ 
mend  tfiat  you  consider  the  document  as  a  whole  at  the  beginning  and  end  of  the  meeting,  and  take  turns  leading 
the  discussion  of  individual  paragraphs  while  you  are  going  through  the  document  from  beginning  to  end. 

The  coordinator’s  final  responsibility  is  to  finish  the  summary  report  and  obtain  signatures  before  the  meeting 
ends.  At  the  conclusion  of  the  meeting,  the  summary  report  should  be  submitted  to  the  meeting  observer. 

Recorder  Responsibilities:  The  recorder  may  be  too  busy  at  first  to  participate  a  whole  lot.  The  recorder  job 
will  switch  at  break  in  order  to  allow  him  or  her  to  participate  more  fully,  and  to  spread  the  burden  of  preparing 
the  final  annotated  document.  Try  to  record  in  such  a  way  that  your  notes  are  open  to  everyone  at  the  meeting, 
and  usable  for  forming  the  final  reports.  The  technique  of  marking,  numbering,  and  writing  detailed  remarks  on 
a  separate  sheet  of  paper  works  reasonably  well,  especially  with  a  single-spaced  document. 

After  the  meeting,  a  final  version  of  your  group’s  annotations  must  be  prepared  as  a  group  document.  The 
recorders  are  responsible  for  submitting  this  final  document  in  the  same  manner  in  which  they  submitted  their 
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individual  documents.  We  recommend  that  you  do  this  right  after  the  meeting,  but  you  have  until  the  next 
regular  clasi  meeting. 
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In  the  style  of  [Fagan76] 


At  the  time  when  individual  comments  arc  landed  in,  your  group  must  choose  a  coordinator,  a  reader/recorder, 
and  individuals  who  will  assume  the  points  of  view  of  author,  client,  and  consumer.  Responsibilities  for  each  of 
these  roles  are  detailed  below.  Under  the  leadership  of  this  newly  selected  coordinator,  the  group  then  chooses  a 
time  for  a  two-hour  group  meeting.  Schedule  this  meeting  within  four  days,  since  your  preparation  for  the 
meeting  must  be  fresh  in  your  minds.  Allow  time  after  the  group  meeting  for  your  recorder  to  prepare  your  final 
document  before  the  due  date. 

Everyone  in  the  group  must  participate  in  the  entire  group  meeting  in  order  to  receive  group  reviewing  points. 
Your  instructor,  or  an  official  representative,  will  observe  the  meeting  and  make  an  audio  recording  of  what  is 
said.  The  meeting  should  take  no  longer  than  two  hours. 

Coordinator  Responsibilities;  The  coordinator's  first  responsibility  is  seeing  that  the  arrangements  for  the 
meeting  are  all  taken  care  of  This  includes  seeing  that  everyone  gets  there  on  time.  In  a  professional  setting, 
the  coordinator  would  also  schedule  a  room,  prepare  copies  of  necessary  materials,  and  see  that  all  the  review 
participants  are  prepared  for  the  meeting.  In  this  case,  your  instructor  will  take  care  of  the  room  and  the  review 
materials.  Any  group  member  who  fails  to  prepare  for  the  meeting  will  already  have  been  eliminated  at  the 
individual  review  submission  time. 

The  coordinator’s  second  responsibility  is  to  act  as  a  meeting  leader.  At  the  group  review  meeting,  the  coor¬ 
dinator  should  obtain  the  group’s  agreement  on  what  must  be  accomplished  in  the  meeting  and  who  will  assume 
what  roles,  and  then  see  that  the  meeting  goals  are  achieved  as  well  as  possible.  This  means  that  the  coordinator 
should  keep  the  proceedings  on  track  and  keep  an  eye  on  the  time  The  coordinator  must  ensure  that  the 
comments  documented  before  die  meeting  are  understood,  merged,  edited,  or  deleted,  in  whatever  manner  the 
group  feels  might  be  helpful.  Whenever  it  is  appropriate,  the  old  comments  should  be  extended,  and  new 
comments  generated.  Your  group’s  final  reports  on  the  software  that  is  being  reviewed  must  state  overall 
strengths  and  weaknesses  and  emphasize  major  issues,  so  be  especially  alert  for  main  themes  throughout  the 
two-hour  period.  If  agreement  cannot  be  reached  on  any  point,  consider  the  most  negative  opinion  to  be  the 
review  outcome,  but  report  that  there  was  disagreement  on  that  point.  The  coordinator  is  also  responsible  for 
deciding  whether  or  not  to  take  a  break. 

The  coordinator  should  not  have  to  use  heavy-handed  or  autocratic  tactics,  since  the  reviewers  are  competent, 
well-prepared  (almost)  professionals.  The  agenda  is  driven  by  the  reader.  Consider  the  document  as  a  whole  at 
the  beginning  and  end  of  the  meeting.  Since  the  author  is  typically  not  very  active  in  this  context,  have  the 
author  and  read  switch  roles  at  some  point. 

The  coordinator  should  submit  the  group’s  summary  report  at  the  conclusion  of  the  group  meeting.  The  coor¬ 
dinator  must  also  see  that  the  recorder  submits  a  final  set  a  detailed  remarks. 

Reader/Recorder  Responsibilities;  The  “reader”  in  an  inspection  is  responsible  for  reading  the  software,  or 
paraphrasing  it,  one  block  (or  paragraph)  at  a  time.  The  reader  notes  any  individually  prepared  remarks  about 
each  block  after  it  is  read.  The  coordinator  then  leads  the  discussion  of  that  block  while  the  reader/recorder 
makes  a  written  record  of  all  the  concerns  of  the  group. 

The  reader/recorder  may  be  too  busy  at  first  to  participate  a  whole  lot.  This  job  may  switch  at  break  in  order  to 
allow  each  reader/recorder  to  participate  more  fully,  and  to  spread  the  burden  of  preparing  the  final  aimotated 
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document.  Try  to  record  in  such  a  way  that  your  notes  are  open  to  everyone  at  the  meeting,  and  usable  for 
forming  the  final  reports.  The  technique  of  marking,  numbering,  and  writing  detailed  remarks  on  a  separate 
si  )f  paper  works  reasonably  well,  especially  with  a  single-spaced  document. 

After  the  meeting,  a  final  version  of  your  group’s  annotations  must  be  prepared  as  a  group  document.  The 
reader/recorder(s)  are  responsible  for  submitting  this  final  document  in  the  same  manner  in  which  they  sub¬ 
mitted  their  individual  documents.  We  recommend  that  you  do  this  right  after  the  meeting,  but  you  have  until 
the  next  regular  class  meeting. 

Author;  The  author  is  responsible  for  knowing  the  application  domain  and  all  technical  detail  that  has  to  do 
with  the  software  under  review.  Reviewers  may  ask  the  author  for  detailed  explanations  of  the  software  or  the 
application  context.  The  author  may  also  ask  the  reviewers  to  clarify  an  issue  that  they  want  to  raise.  With  a 
small  group  where  the  author  is  only  “playing  a  role,”  the  author  may  double  as  reader,  and  possibly  even 
recorder. 

Client:  The  client  is  the  person  responsible  for  the  software  elements  upon  which  the  software  under  review  is 
based,  or  generally  a  software  user.  The  person  who  assumes  this  role  should  be  concerned  with  the  validity  and 
completeness  of  the  software  that  is  being  reviewed. 

Consumer;  The  consumer  is  the  person  who  will  implement  the  next  step  in  the  software  develofxnent  life 
cycle.  This  person  should  be  concerned  with  the  completeness,  clarity,  and  verifiability  of  the  software  under 
review.  In  student  inspections,  this  person  can  also  play  the  role  of  tester. 
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This  technique  may  be  used  when  other  forms  of  group  interactions  arc  difficult  to  arrange,  or  for  a  controlled 
study. 

At  the  time  when  individual  review  comments  arc  due,  schedule  a  two-hour  period  when  you  will  work  on  your 
group  review  under  the  supervision  of  your  instructor.  During  this  time,  you  will  produce  a  review  that  reflects 
the  combined  insight  of  your  group,  and  its  effect  on  you.  Your  instructor  will  provide  you  with  a  master  copy 
of  the  review  document  into  which  he  has  merged  the  remarks  of  your  group  members.  You  will  have  two  hours 
to  react  to  your  group’s  combined  comments  and  prepare  a  summary  group  evaluation  report  (a  form  is  attached 
to  these  directions).  You  will  not  have  access  to  a  terminal  during  this  group  review  time. 

Your  role  during  this  type  of  a  group  software  review  is  really  that  of  a  review  coordinator,  but  you  have  the 
option  to  add  to  the  final  review  output  in  any  way  that  you  feel  might  be  helpful.  Since  you  are  producing  the 
only  final  output  from  the  effort  of  the  entire  group,  do  your  best  to  see  that  the  software  review  goals  are 
achieved  as  well  as  possible.  Keep  your  thoughts  on  track  and  keep  an  eye  on  the  time.  Make  sure  that  you 
understand  the  individual  comments,  then  merge,  edit,  or  delete  remarks  in  whatever  manner  appears  to  be 
helpful.  The  old  comments  should  be  extended,  and  new  comments  generated  whenever  possible.  Your  final 
reports  on  the  software  that  you  are  reviewing  must  also  state  overall  strengths  and  weaknesses,  and  emphasize 
major  issues,  so  be  especially  alert  for  main  themes  throughout  the  two  hour  period.  If  the  individual  comments 
disagree  in  an  important  way  that  you  do  not  feel  you  can  resolve  by  yourself,  base  your  summary  report  on  the 
most  negative  remark,  but  report  that  there  was  disagreement  on  that  point. 

After  your  two  hours  are  over  (or  when  you  are  ready),  submit  your  summary  report  and  a  copy  of  your 
handwritten  comments  to  your  instructor,  who  will  make  a  photocopy  and  give  you  back  your  originals  before 
you  leave.  You  have  until  the  next  regular  class  meeting  to  make  your  annotated  changes  in  the  master  docu¬ 
ment  for  your  version  of  your  group’s  remarks.  Submit  these  final  “group  remarks’’  in  the  same  way  in  which 
you  submitted  your  individual  remarks. 
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Computer-Mediated  Group  Meeting 
Using  Honeywell  CB  Software^ 


At  the  class  meeting  when  individual  comments  are  due,  choose  a  coordinator  and  two  recoiders. 
(Responsibilities  for  each  of  these  roles  are  detailed  below.)  Under  the  leadership  of  your  coordinator,  arrange 
two  times  for  a  90-minute  CB  meeting,  one  in  an  optimistic  time  frame  for  completing  your  meeting  prepara¬ 
tions,  and  one  in  a  pessimistic  time  frame.  The  point  is  that  you  want  to  have  the  meeting  as  soon  as  you  are 
ready,  but  getting  ready  may  take  longer  than  you  plan.  You  need  to  get  your  scheduling  done  in  a  face-to-face 
mode,  because  arranging  a  meeting  time  through  MAIL  can  be  very  difficult.  The  total  CB  time  should  not  go 
over  90  minutes,  plus  whatever  time  you  spend  practicing. 

You  will  be  using  electronic  mail,  CB,  and  additional  software  tools  to  help  you  interact  through  the  CP6.  The 
idea  of  computer-mediated  group  interaction  is  to  allow  you  to  work  in  a  geu^’raphi rally  dispersed  mode.  You 
should  decide  on  non  face-to-face  work  sites  to  get  a  feeling  for  this  mode  of  group  interaction.  If  there  is  a 
problem  with  this  ask  your  instructor  for  help  (through  electronic  mail?).  If  you  ever  do  find  yourselves  talking 
face-to-face,  or  working  where  you  can  see  each  other,  please  make  a  note  of  what  happened  in  your  time  report. 

In  preparation  for  the  group  CB  session,  each  group  member  must  obtain  a  printed  copy  of  the  group’s  combined 
remarks.  This  document  will  contain  the  merged  and  numbered  comments  of  your  entire  group.  Your  instructor 
will  prepare  this  document  as  soon  as  possible,  and  adjust  its  attributes  so  that  only  the  members  of  your  group 
can  access  it;  updates  may  only  be  made  after  your  meeting,  and  then  only  by  your  group. 

Each  individual  should  mark  his  or  her  printed  copy  of  the  group’s  remarks  with  an  evaluation  of  the  helpfulness 
of  each  remark,  according  to  the  following  rating  codes. 

5  Excellent,  absolutely  es.sential  remark 
4  Good  remark,  the  group  should  keep  it 
3  Marginally  helpful  remark 

2  Consider  deleting  this  remark 

1  Delete  this  remark 

*  Discuss  this  remark  in  the  group  meeting. 

Cnnn  Combine  with  remark  number  nnn. 

Then,  run  VOTE  (executable  code)  and  enter  your  response  to  each  remark.  Your  coordinator  will  prepare  a 
meeting  agenda  from  the  rcsailts  of  your  group’s  VOTEs,  together  with  any  mail  comments  that  you  send  him  or 
her.  This  agenda  will  be  distributed  to  you  through  mail. 

If  any  group  member  is  unacceptably  slow  in  working  on  this  task,  the  group  coordinator  has  the  responsibility 
to  declare  that  individual  to  be  off  the  team  (resulting  in  a  score  of  zero  for  that  individual’s  group  review  part  of 
the  assignment),  or  work  something  out  so  that  the  group  is  not  hurt  by  that  individual.  Any  group  member  who 
does  not  participate  in  the  CB  session  will  receive  a  score  of  zero  for  the  group  review.  If  the  coordinator  is  not 
doing  his  or  her  job  well  enough,  call  in  your  instructor  as  soon  as  possible. 

Your  instructor  will  produce  a  computer- readable  copy  of  your  CB  session,  which  your  recorders  will  need. 


^The  CB  program  simulates  a  citizen’s  band  radio  communication  group, 
to  each  other. 


Multiple  online  users  may  monitor  and  broadcast  messages 
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Make  sure  that  your  instructor  knows  when  the  meeting  will  happen,  and  that  he  is  maintaining  a  machine- 
readable  copy  of  your  dialogue  before  you  begin.  Please  submit  copies  of  all  your  project  mail  (through  elec¬ 
tronic  mail,  of  course)  to  complete  your  group  project  records. 

CB  Protocol:  The  CB  software  could  be  a  lot  more  helpful,  but  it  can  be  functional.  A  few  ground  rules  for 
group  interaction  should  help  a  lot  The  coordinator  is  in  charge.  When  you  want  to  say  something,  send 
The  coordinator  will  give  you  control  of  the  communication  medium  until  you  have  your  say.  End  your  message 
with  "#”  (or  a  10-4?).  Practice  this  procedure  before  you  begin  your  90-minute  meeting  in  earnest.  The 
coordinator  may  modify  this  protocol,  but  let  your  instructor  know  what  you  do  and  why. 

Coordinator  Responsibliltlea:  The  coordinator’s  first  responsibility  is  seeing  that  tlie  arrangements  for  the 
meeting  are  all  taken  care  of.  This  includes  all  scheduling  arrangements,  and  deciding  what  to  do  if  something 
does  not  go  as  planned. 

The  coordinator’s  second  i^ponsibility  is  to  act  as  a  leader  for  the  entire  task.  This  includes  writing  an  agenda, 
seeing  that  everyone  prints  and  reads  a  copy  of  the  agenda,  and  controlling  the  CB  session.  The  coordinator 
must  keep  an  eve  on  the  time  and  keep  the  discussion  on  track.  You  will  probably  mn  out  of  time,  so  break  in  at 
an  appropriate  time  and  ask  for  everyone’s  overall  impressions  before  you  run  out  of  CB  time.  You  must 
prepare  a  copy  of  the  Summary  Report  from  the  CB  session,  so  make  sure  that  you  have  everyone’s  remarks  in 
the  CB  dialogue.  Your  group’s  final  report  on  the  software  that  is  being  reviewed  must  state  overall  strengths 
and  weaknesses  and  emphasize  major  issues,  so  be  especially  alert  for  main  themes  throughout  the  90-minutes 
of  CB  interaction.  If  agreement  cannot  be  reached  on  any  point,  consider  the  most  negative  opinion  to  be  the 
review  outcome,  but  report  that  there  was  disagreement  on  that  point. 

Recorder  Responsibilities:  After  the  meeting,  a  final  version  of  the  review  annotations  must  be  prepared  as  a 
group  document.  The  recorders  are  responsible  for  submitting  this  final  document  The  deadline  for  this  is  the 
time  of  the  next  regular  class  meeting.  Prepare  this  document  by  editing  a  copy  of  your  group’s  remarks 
according  to  the  results  of  VOTE  and  the  machine-readable  record  of  die  CB  session. 

Ask  for  whatever  clarification  you  need  during  the  CB  session(s).  When  you  write  the  final  group  comments,  if 
something  is  still  unclear,  or  the  group  is  not  in  agreement,  say  so,  and  record  the  most  negative  group  opinion. 
The  recorders  should  not  change  the  substance  of  what  is  decided  in  the  group  review  meeting. 

After  the  group  interaction,  all  comments  are  from  the  group,  not  individuals,  so  do  a  global  edit  that  removes 
the  initials  of  the  original  author  of  the  comment. 
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^oftware  Inspection  Summary  and  Evaluation  Report 

Form 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Forms  for  reporting  of  outcomes  of  a  software  technical  review. 

Objectives: 

•  In  actual  practice:  to  provide  project  management  with  a  means  of  control  over  the  project  devel¬ 
opment  process  and  to  provide  the  software  developer  with  a  list  of  specific  issues  raised  in  the 
technical  review. 

•  In  a  situation  where  the  primary  goal  is  to  learn  about  software  technical  reviews: 

•  to  ensure  that  students  perform  aU  the  specified  technical  review  procedures. 

•  to  implement  summary  and  evaluation  reporting,  an  essential  part  of  the  software  technical 
review  process. 

•  to  force  students  to  move  beyond  specific  details  to  evaluate  and  summarize  on  a  more  gen¬ 
eral  level. 

Prerequisites:  Students  must  be  familiar  with  the  technical  review  process  and  the  procedure  for  work  within 
each  group. 

Pf  ure:  Each  group  receives  a  copy  of  the  Summary  Report  form,  along  with  a  compilation  of  the  com¬ 
ment..  of  each  member  of  the  group.  The  instructor  must  emphasize  that  the  form  contains  an  essential  agenda. 
It  is  the  review  leader’s  respoasibility  to  follow  that  agenda  and  submit  the  group’s  final  report.  The  instructor 
should  stress  that  each  item  on  the  agenda  must  be  conscientiously  completed.  The  review  leader  should  com¬ 
plete  the  form,  but  may  delegate  the  task  of  writing  the  Summary  of  Findings  section  to  a  group  member. 

Two  items  on  tlte  form  tire  particularly  difficult:  a  summary  of  findings  and  an  evaluation  decision.  Both  items 
require  group  consensus  on  broad  issues.  The  instructor  should  emphasize  tihat  participants  in  a  review  should 
be  concerned  not  only  with  individual  issues  relative  to  specific  points  in  the  text  of  the  software,  but  also  with 
an  overall  evaluation  of  the  quality  of  the  software.  The  evaluation  must  be  supported  by  a  cogently  worded 
summary.  Review  leaders  should  be  instructed  to  assume  the  respionsibility  for  allocating  adequate  time  to 
agenda  items  (3)  through  (6).  A  minimum  of  twenty  minutes  at  the  end  of  a  two-hour  meeting  is  recommended. 

The  review  leader  is  responsible  for  focusing  the  group’s  effort  on  the  software  under  review.  If  an  important 
issue  raised  in  the  review  is  not  specifically  or  solely  a  defect  in  the  software,  the  instructor  may  permit  tire 
group  to  attach  a  Related  Issues  Report  as  a  separate  item.  Examples  of  related  issues  include  notes  about  tlie 
meeting  facilities  or  group  procedures  for  the  review.  Some  experts  (e.g.,  Collofello  in  CM-3)  recommend  that 
related  issues  not  be  rtporuid  because  tlie  possibility  of  reporting  related  issues  may  distract  the  group  from  its 
primary  concern — the  software  itself.  The  review  leader  should  not  allow  extended  discussion  of  any  issue;  the 
point  of  a  group  review  meeting  is  to  raise  issues,  not  resolve  tiiem.  This  is  a  particularly  true  of  related  issues! 

Evaluation:  Technical  review  groups  need  specific  feedback  on  their  performance,  both  in  terms  of  what  they 
did  or  did  not  report  and  how  well  they  re.ported  it.  However,  the  instructor  may  find  it  difficult  to  quantify  how 
well  each  group  completed  its  summary  report.  Realistically,  the  instructor  can  only  grade  (1)  the  completeness 
of  the  form  and  (2)  the  quality  of  the  summary  of  findings  and  the  related  issues  list  (if  one  exists).  The 
summary  report  should  constitute  one-fifth  to  one-tliird  of  the  group’s  grade.  Additional  points  may  Ije  awarded 
to  th"  students  who  sign  as  review  leader  or  recorder.  If  the  group  feels  that  one  of  its  members  did  especially 
we  poorly  in  the  group  meeting,  this  feeling  can  be  noted  as  a  “related  issue”  and  used  as  a  basis  for 
awaruing  a  higher  or  lower  score  to  that  individual. 
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The  person  who  evaluates  tire  quality  of  a  software  technical  review,  based  on  the  data  reported  on  the  attached 
form,  should  be  sensitive  to  how  the  group  allots  its  time  to  each  agenda  item  and  what  non-agenda  items 
receive  significant  group  attention.  The  groups  are  likely  to  spend  most  of  their  energy  on  item  (2)  of  the  agenda, 
examining  the  line-by-line  concerns  of  the  group.  However,  it  is  important  to  learning — and,  in  practice,  it  is 
important  to  the  project — for  the  technical  review  group  to  spend  a  reasonable  amount  of  its  energy  on  a  broad 
view  of  the  itemized  findings.  For  this  reason,  the  instructor  may  want  to  ask  the  group  to  enter  the  time  of 
completion  of  each  agenda  item,  rather  than  simply  checking  each  one  off. 

Commentary:  Students  tend  to  focus  on  the  details  of  the  group’s  annotations  of  the  software  that  has  been 
reviewed.  TTie  group  must  have  strong  leadership  to  ensure  that  each  item  on  tlie  agenda  receives  adequate 
attention.  Students  are  likely  to  be  surprised  at  the  differences  between  the  group’s  repoit  and  their  individual 
input,  and  between  the  instructor’s  “grading  key’’  and  the  group  report.  These  differences  offer  significant 
opportunities  for  learning,  so  the  feedback  process  is  important.  The  best  groups  achieve  synergistic  output,  in 
which  the  outcome  of  the  group  meeting  includes  concerns  not  reponed  in  the  individual  preparation  of  any 
group  member  for  the  group  meeting. 

The  following  form  is  one  of  several  widely  used  forms.  (Fagan76]  has  published  forms  which  have  been  used 
at  IBM  sites;  these  forms  include  counts  of  defects  in  three  categories.  The  author  of  this  document  selected  a 
form  that  emphasizes  agenda  and  minimizes  counting  and  categorization.  Error  counts  were  omitted  because  of 
the  automated  processes  used. 

Note:  Similar  foims  appear  in  [Freedman82]  and  [YourdonSS). 
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Software  Inspection  Summary  and  Evaluation  Report 


Project _ Date _ 

Review  leader _ Start  Time _ Stop  Time 


Agenda  (Enter  the  time  when  completed) 

_ 1.  Agree  to  functional  roles  (review  leader,  recorder,  and  (optional)  reader);  method  of  considering 

different  points  of  view  (user,  producer,  client,  tester,  quality  assurance);  review  goals;  and  procedures. 

_ 2.  Review  merged  comments:  filter,  combine,  refine,  and  generate  new  comments. 

_ 3.  Summarize  major  issues  which  have  been  raised  by  the  group.  List  these  major  issues  in  the 

Summary  of  Fmdings. 

_ 4.  Decide  on  the  overall  state  of  the  software  and  report  that  decision  on  this  form.  Verify  that  the 

evaluation  decision  is  supported  in  the  Summary  of  Findings. 

_ 5.  Attach  to  this  report  all  documented  review  outcomes,  including  the  group’s  edited  annotations  for 

the  original  software.  If  there  is  a  Related  Issues  List,  note  that  fact  under  the  Summary  of  Findings. 

_ 6.  Sign  this  report. 


Summary  of  Findings 


Evaluation  Decision 

_ Accept  as  is  (OK). 

_ Accept  after  minor  revisions  (OKR). 

_ Unacceptable;  revise  and  plan  another  review 

(NOK). 


Signatures 
(Note  any 
special  role) 
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John  A.  Cross 

'ncliana  University  of  Pennsylvania 


Description:  Related  Issues  Report  Form. 

Purpose:  To  provide  a  means  for  reporting  significant  issues  that  require  the  involvement  of  others,  that  cannot 
be  resolved  solely  by  the  producer  of  the  .software  artifact. 

Prerequisites:  To  u.se  this  form,  students  must  know  what  is  considered  a  related  issue. 

Procedure:  A  brief  explanation  is  included  with  the  form.  Students  should  be  aware  that  this  form  is  optional. 

Evaluation:  If  this  form  is  used,  the  data  reported  should  be  considered  in  die  evaluation  of  the  >:,roup’s 
summary  report. 

Commentary:  Related  issues  should  be  reported,  but  they  should  not  be  discussed.  TTie  review  meeting  must 
concentrate  on  the  software  under  review — the  coordinator  (moderator)  must  not  allow  the  attention  of  the  group 
to  be  diverted  to  related  issues.  Sample  related  issues  are  given  on  the  form. 
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RELATED  ISSUES  REPORT  FORM 


Project _ 

Coordinator 
Date _ 


GENERAL  GUIDELINES 


This  form  may  be  used  to  report  any  concerns  which  are  raised  in  a  software  technical  review,  but  are  not  the 
appropriate  or  sole  responsibility  of  the  software  author.  Related  issues  generally  fall  into  one  of  the  following 
general  categories; 

•  There  is  a  defect  in  an  artifact  upon  which  the  software  under  review  was  based.  For  example,  a 
review  of  code  may  turn  up  an  error  in  the  low-level  design  of  that  code. 

•  Standards  are  in  some  way  inadequate. 

•  Facilities,  procedures,  or  participants  are  worthy  of  some  special  formal  report.  For  example,  it 
should  be  reported  if  too  many  participants  were  scheduled  for  the  review. 


NUMBERED  LIST  OF  RELATED  ISSUES 
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Categories  for  Defects  in  Software  Technicai  Reviews 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Topic:  Selecting  a  category  for  each  software  defect  which  is  reported  by  a  software  technical  review. 

Objectives:  Requiring  reviewers  to  categorize  each  defect  may  significantly  increase  the  difficulty  of  reporting 
defects.  The  objectives  of  the  process  must  be  cleariy  defined  and  the  data  must  be  monitored  to  assure  tliat  the 
objectives  are  being  met  The  benefits  of  reporting  categories  must  justify  the  costs.  The  reasons  for  requiring 
software  reviewers  to  categorize  each  defect  include  the  following: 

•  To  provide  data  which  can  be  analyzed  to  determine  facts  about  the  software  technical  review 
process. 

•  To  structure  the  documentation  of  defects  and  the  corresponding  thought  processes  which  underlie 
that  activity. 

•  To  increase  the  amount  of  detail  in  the  documentation  of  each  defect. 

Prerequisite  Knowledge:  If  the  software  technical  review  process  includes  categories  for  each  issue  raised, 
reviewers  must  have  a  simple,  clear  set  of  standard  guidelines.  Reviewere  must  understand  that  the  data  they 
provide  has  significant  value  to  their  organization  even  though  categorizing  the  issues  is  difficult  and  inexact  at 
best.  Reviewers  must  also  accept  that  “management”  (in  learning  situations,  the  instmetor)  understands  that  the 
categories  may  not  be  an  exhaustive  list  of  every  type  of  issue  that  is  appropriate  to  raise  in  a  review. 

Procedure:  When  providing  a  set  of  categories,  the  instructor  should  consider  providing  examples  of  issues  for 
each  category — issues  which  may  be  categorized  in  different  ways  depending  on  the  point  of  view  of  the 
reviewer — and  examples  of  the  uses  to  which  these  data  might  be  put  by  the  organization  requiring  adherence  to 
a  specific  set  of  standardized  categories. 

Reviewers  should  record  categories  within  the  text  describing  the  defect.  For  example,  the  IBM  procedure  of 
categorizing  defects  as  missing,  extra,  or  wrong  fFagan76I  can  be  implemented  in  an  online  reporting  context  by 
prefixing  tlie  text  of  each  defect  by  an  M,  E,  or  W.  Similarly,  the  [Bell76]  categories,  which  are  listed  here,  might 
be  encoded  with  the  error  category  enclosed  in  square  brackets. 

The  person  acting  as  recorder  for  the  group  has  the  final  responsibility  of  assigning  categories.  Whenever  a 
group  of  reviewers  are  involved,  it  is  likely  that  different  reviewers  will  assign  different  categories  to  the  same 
defect  (these  data  are  considered  unreliable).  In  order  to  avoid  requiring  the  reviewers  to  resolve  this  disagree¬ 
ment,  the  instmetor  may  want  to  allow  recorders  to  assign  more  than  one  category  to  a  single  defect.  For 
example,  the  following  defect  might  be  reported  by  a  group  using  the  [BeL176|  categories: 

[1-08,  6-01]  Thtt  coxoputation  of  interest  earnings  is  based  upon  average 
daily  balance,  but  the  precision  of  this  average  daily  balance  has  not 
been  specified. 

The  review  group  should  not  spend  time  choosing  the  best  category;  that  is  why  the  task  is  assigned  to  the 
recorder.  Categories  which  repeatedly  overlap  may  indicate  that  the  list  needs  to  be  refined  or  explained  better. 

Evaluation:  It  is  difficult  to  assign  scores  to  this  activity  in  a  classroom  setting.  Summary  statistics  should  oe 
computed,  with  an  automatic  process  if  possible.  When  the  [Bell76]  categories  were  used  by  the  author  of  this 
document,  statistical  analysis  of  the  results  showed  that  different  groups  of  students  focused  on  different  groups 
of  categories.  Points  may  be  subtracted  or  added  to  the  group  score  for  group  oversight,  error,  or  helpful  actions 
relative  to  categories.  Individual  students  should  not  be  penalized  unless  their  categories  are  incomplete  or 
grossly  incorrect 

Commentary:  The  categorization  process  appears  to  be  helptul  for  students,  but  the  data  have  low  reliability. 
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The  author’s  students  found  the  [Bell76]  categories  complex  and  difficult  to  apply.  The  [Fagan76]  categories  arc 
even  more  complex  because  three  separate  categorizations  are  required.  The  following  subset  of  the  [Fagan76] 
categories  is  attractive  because  it  is  simpler  and  may  be  more  reliable: 

Alternative  categories: 

M,  E,  W  for  Missing,  Extra,  or  Wrong 
MAJ  for  Major  or  MIN  for  Minor 

Each  defect  must  be  categorized  both  as  M,  E,  or  W,  and  as  MAJ  or  MIN. 
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Software  Specifications;  Categories  for  Remarks 


ERROR 

CATEGORY  PROBLEM  DESCRIPTION 


*1-00  Missing/Incomplete/Inadequate 
1  -01  Criteria  for  a  system  decision  missing  or  inadequate 
1  -02  Interface  characteristics  missing 

1  -03  Accuracy  or  precision  criteria  missing 
1  -04  Description  of  context  inadequate 
1  -05  Processing  specification  missing 
1  -06  Error  recovery  specification  missing 

1-07  Missing  emphasis 
1  -08  Data  or  process  validation  criteria  missing 

1-09  Acceptance  criteria  missing 

1- 10  Data  specification  missing 

2- 00  Inconsistent/Incompatible 

2- 0 1  Two  or  more  specifications  disagree 

2-02  Incompatible  with  existing  standards 

2- 03  Incompatible  with  existing  systems 

3- 00  Unclear 

3- 01  Terms  or  acronyms  need  to  be  defined 

3-02  Ambiguous  woixling 

3-03  Muddled  writing 

3- 04  Specification  doesn’t  make  sense 

4- 00  Extra 

4- 01  Outside  of  the  scope  of  this  project 

4-02  Unnecessary  detail 

4-03  Redundant  or  wordy 

4- 04  Overly  restrictive  (includes  specifications  which 

are  stated  at  too  low  a  level) 

5- 00  Incorrect  form 

5- 01  Typographical  or  spelling  error 

5-02  Writing  error 

5-03  Word  processing  error 

5- 04  Violation  of  standards 

6- 00  Incorrect  technical  detail 

6- 01  Specified  processing  inaccurate  or  imprecise 

6-02  Specified  processing  inefficient 

6-03  Specification  not  testable 

6-04  Specification  not  modifiable 

6-05  Ciescription  of  problem  or  context  incorrect 
6-06  Technical  error 

7- 00  General  remarks 


*  Derived  from  a  list  in  Bell,  T.  E.,  and  'fhayer,  T.  A.,  “Software  Specifications;  Are  They  a  Problem?”  Proc. 
IEEE! ACM  Second  Inti.  Conf.  on  Software  Engineering,  October  1976. 
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Checklists  for  Reviewers 

John  A.  Cross 

Indiana  University  of  Pennsylvania 


Description:  Checklists  for  use  in  software  technical  teviews.  The  following  checklists  were  derived  from 
[Freedman82],  [Bell76],  and  [Fagan76]. 

Purpose:  To  be  used  by  reviewers  to  ensure  that  a  specific  set  of  possible  defects  is  considered. 

Procedure:  These  checklists  may  be  used  by  student  reviewers. 

Commentary:  The  appropriateness  of  any  checklist  should  be  carefully  considered  by  the  instructor.  Major 
considerations  include: 

•  Is  the  particular  checklist  suitable  for  the  software  under  review,  the  reviewers’  abilities,  and  the 
procedures  being  followed? 

•  What  defects  arc  not  covered  by  the  list?  Teaching  note:  students  should  know  that  any  a  priori  list 
of  possible  defects  can  have  the  counterproducti  'e  result  of  blinding  reviewers  to  defects  that  do  not 
fit  any  of  the  categories  on  the  list. 

•  Do  the  students  understand  the  listed  items  and  how  to  inspect  for  them?  Teaching  note:  examples 
of  the  defects  and  practice  in  defect  detection  arc  recommended.  The  sample  examination  materials 
(p.  15)  provide  an  indication  of  the  type  of  materials  that  can  be  used. 

•  Is  it  appropriate  to  have  reviewers  categorize  the  defects  they  note  in  their  reviews?  Teaching  note: 
the  IBM  procedure  of  categorizing  each  defect  as  something  which  is  missing,  extra,  or  wrong  is 
easy  to  explain  to  students,  but  the  effect  of  anj  categorizing  procedure  .should  be  carefully  consid¬ 
ered.  Individual  reviewers  must  spend  time  an.  >  thought  when  they  decide  on  a  category  and  dis¬ 
crepancies  between  individual  reviewers  must  be  resolved  during  group  interaction.  (If  the  IBM 
procedure  is  used,  it  may  be  helptul  to  add  a  category  for  style  annotations,  which  results  in  an 
acronym  of  MEWS  for  the  complete  set  of  catCtOries.) 

The  following  checklists  are  included  in  response  to  p(  pular  demand,  not  because  they  are  highly  recommended. 
Contributions  of  helpful  checklists  are  needed.  A  he.pful  checklist  must  have  proven  validity  in  a  significant 
breadth  of  technical  reviewing  contexts. 
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General  Software  Reviewing  Checklist 


Lists  of  attributes  of  quality  software  have  been  published  in  software  engineering  literature  (c.g.,  [Ek)0hm76] 
and  [Freedman82)).  The  following  checklist  is  an  organized  collection  of  criteria  for  quality  software.  By  itself, 
this  list  may  be  too  general  for  a  thorough  software  review,  but  it  can  provide  a  helpful  foundation. 

Directions:  Mark  each  point  as  “Not  Applicable"  (NA),  “Acceptable  As  Is"  (OK),  “Correct  with  Minor  Revi¬ 
sions  Needed"  (OKR),  or  “Unacceptable"  (NOK).  Cite  specific  problems  by  referring  to  numbered  remarks 
and/or  a  specific  point  in  the  document  being  reviewed. 


(1)  CLARITY:  The  software  must  be  clear,  unambi{tuous,  and  precise. 


_ (2)  CORRECTNESS:  The  software  must  satisfy  its  specificatioos,  and  the  final  product  must  satisfy  its 

requirements. 


_ (3)  COMPLETENESS:  All  required  fimctions  are  fully  implemented.  j'UI  software  is  necessary  iind 

sufficient. 


_ (4)  CONSISTENCY:  Design  and  implementation  techniques,  as  well  as  notations,  are  uniform  and  in 

agreement  with  standards. 


_ (5)  TESTABILITY:  All  system  functions  can  be  empirically  verified  witliout  inordinate  cost.  The  soft¬ 
ware  is  safe. 


_ (6)  LEARNABILITY:  The  effort  required  to  learn  how  to  use  the  software  is  not  unreasonable.  Seriouo 

misconceptions  are  unlikely. 


(7)  USABILITY:  The  system  is  reasonably  easy  for  its  expected  users  to  use. 


_ (8)  ROBUSTNESS:  Computer  programs,  and  the  systems  which  they  support,  must  continue  to  function 

according  to  specifications,  even  when  small  errors  occur  during  their  use.  The  system  must  appropriately 
respond  to  and  recover  from  catastrophic  abnormalities  in  its  operating  context. 


(9)  ERROR-TRAPPING:  Computer-based  systems  must  detect  errors  and  deal  with  them  appropriately. 


_ (10)  MODIFIABILITY:  It  should  be  possible  to  change  the  behavior  or  operating  context  of  the  software 

component  of  computer-based  systems  withtii  reasonable  limits  and  without  unacceptable  cost. 


_ (11)  OTHER  DEFECTS:  This  category  includes  any  additional  deficiencies  in  the  ways  in  which  the 

software  meets  its  requirements. 
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Software  Requirements  Reviewing  Checkiist 


A  software  requirements  document  must  state  why  a  system  is  needed.  It  should  focus  on  defining  rVl^  problem 
and  the  context  in  which  a  solution  must  function.  Software  requirements  include  everything  which  is  r  quired 
for  a  system  to  be  useful.  Optional  system  features  and  ideal  system  functionality  may  be  included,  the  extent 
that  it  might  influence  subsequent  system -related  decisions.  The  goal  of  a  software  requirements  document  is  to 
make  a  complete  statement  of  a  problem,  together  with  known  constraints  on  a  solution.  When  they  are  relevant 
to  the  requirements  of  a  software  system,  the  requirements  document  must  discuss  the  points  listed  below 
([Abbob86]  and  ISommerville85]). 

Directions:  Mark  each  point  as  "Not  Applicable"  (I'^A).  “Acceptable  As  Is"  (OK),  “Correct  with  Minor  Revi¬ 
sions  Needed"  (OKR),  or  “Unacceptable"  (NOK).  Cite  specific  problems  by  referring  to  numbered  remaiks 
and/or  a  specific  point  in  the  document  Iwing  reviewed. 


_ (1)  TTie  general  rationale  fora  system  development  project — why  a  system  should  be  developed. 

_ (2)  The  conceptual  model  on  which  ihe  the  requirements  are  based — how'  the  system  will  be  used,  together 

with  other  systems  and  procedures  that  wil.'  interface  with  the  planned  system. 

_ (3)  Data  items  that  the  system  must  handle,  e.g..  entities,  attributes,  and  relations. 

_ (4)  Volume  estimates  for  the  data  that  the  system  must  handle. 

_ (5)  Integrity  constraints  that  the  system  must  enforce,  e.g.,  access  limitations  and  error-handling  require¬ 
ments. 

_ (6)  Reliability  and  availability  coastraints,  e.g.,  cons<jqucnces  of  system  failure  and  times  when  the  system 

will  be  used. 

_ (7)  Legal  constraints,  e.g.,  privacy  requirements. 

_ (li)  The  knowledge  and  .skill  of  system  u.sors,  e.g.,  how  frequently  the  typical  user  will  interact  with  the 

s}'!item. 

_ (9)  Data  processing  functions  that  the  system  s.hould  perform,  together  with  information  needs  of  the  user, 

e.g.,  which  categories  of  system  entities  must  be  summed  for  reports. 

_ (10)  Hai'dware  and  software  constraints,  e.g.,  constraints  on  the  peripheral  devices  or  programming  lan¬ 
guage. 

_ (11)  Response  time  requirements  and  expected  system  loading. 

_ (12)  Modifications  that  might  be  required  later,  e.g.,  likely  changes  in  the  hardware  or  software  environ¬ 
ment. 

_ (13)  Details  of  standard  form,  e.g.,  title,  author,  date,  index,  glossary  of  terms,  local  standards. 
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Software  System  Specification  Checklist 


A  system  specification  should  focus  on  what  a  system  does  to  meet  the  system  requirements.  Only  external 
behaviors  are  of  direct  concern.  A  software  system  specification  must  describe  these  behaviors  in  such  a  way 
that  a  system  can  be  designed,  built,  and  verified  according  to  them.  Thus,  there  are  really  two  verifications 
involved: 

•  Does  the  specified  system  meet  the  requirements? 

•  Does  the  actual  system  meet  the  specifications? 

A  third  demand  is  often  placed  on  system  specifications:  a  user  or  client  may  want  to  use  the  system  specifi¬ 
cations  to  decide  whether  or  not  to  build  a  system  or  to  determine  which  of  several  competing  specifications  to 
implement.  The  user  or  client  may  also  want  to  modify  the  specifications.  These  uses  for  system  specifications 
require  a  system  specification  statement  that  provides  a  basis  for  system  understanding,  verification,  and  detailed 
design.  The  General  Software  Reviewing  Checklist  is  especially  pertinent  to  system  specifications.  Additional 
criteria  are  listed  below. 

Directions:  Mark  each  t>oint  as  “Not  Applicable”  (NA),  “Acceptable  As  Is”  (OK).  “Correct  with  Minor  Revi¬ 
sions  Needed”  (OKR),  or  “Unacceptable”  (NOK).  Cite  specific  problems  by  referring  to  numbered  remarks 
and/or  a  specific  point  in  the  document  being  reviewed. 


_ (1)  Is  the  specification  clear,  precise,  and  unambiguous  for  all  appropriate  audiences? 

_ (2)  Does  the  specification  state  what  the  system  does  to  meet  all  of  the  systeir.'  requirements? 

_ (3)  Docs  the  specification  provide  an  adequate  basis  for  design  or  direct  implementation? 

_ (4)  Are  user  procedures  specified? 

_ (.5)  Are  installation,  administrative,  and  operations  procedures  specified? 

_ (6)  Are  liehaviors  specified  for  all  system  interfaces? 

_ (7)  Is  there  a  specification  of  the  total  system  supported  by  the  .software? 

_ (8)  Arc  system  object  types  and  instances  specified?  (“Type”  includes  constraints  on  value.) 

_ (9)  Are  system  data  storage  and  processing  limits  s,oecified? 

_ (10)  Arc  the  system  hardware  and  .software  enviroranents  specified? 

___  (11)  Arc  response  time  and  system  loading  constraints  specified? 

_ (12)  Are  exceptions  and  anomalous  inputs  defined? 

_ (13)  Are  reliability,  error-handling,  and  system  backup  procedures  specified? 
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mented. 


74 


Dran  For  Public  Review 


SEI-SM-3-1.0 


The  Software  Technical  Review  Process 


Bibliography 


Qulrk85 

Quirk,  W.  J,  cd.  Verification  and  Validation  of  Real- 
Time  Software.  Berlin:  Springer-Verlag,  1985. 

This  text  concentrates  on  testing  techniques  for  real¬ 
time  software.  The  utilization  of  review  processes 
is  also  described.  The  emphasis  of  these  review 
processes  is.  however,  not  unique  to  real-time  soft¬ 
ware  and  very  little  insight  into  reviewing  real-time 
systems  as  opposed  to  other  types  of  systems  can  be 
obtained  from  this  text. 

Remus79 

Remus,  H.,  and  S.  Zilles.  “Prediction  and  Mtinage- 
ment  of  Program  Quality."  IEEE  Proceedings  of  the 
Fourth  International  Corrference  on  Software 
Engineering.  Silver  Spring,  MD:  IEEE  Computer 
Society  Press,  Sept.  1979,  341-350. 

Abstract:  Techniques  such  as  design  reviews,  code 
inspections,  and  system  testing  are  commonly  being 
used  to  remove  defects  from  programs  as  early  as 
possible  in  the  development  process.  The  objective 
of  the  authors  is  to  demonstrate  that  predictors  can 
be  devised  which  tell  us  how  well  defects  are  being 
removed  during  the  defect  removal  process. 

The  paper  presents  statistical  techniques  for  estimat¬ 
ing  the  number  of  errors  remaining  in  a  product 
based  on  data  collected  from  reviews.  .Approaches 
for  evaluating  reviews  and  the  relationship  of 
various  reviews  to  each  other  and  to  testing  are  also 
described. 

Remus83 

Remus,  H.  "Integrated  Software  Validation  in  the 
View  of  Inspections/Reviews.”  Software  Valida¬ 
tion,  Inspection-Testing-Verification-Alternatives: 
Proceedings  of  the  Symposium  on  Software 
Validation.  Amsterdam:  North-Holland,  Sept.  1983, 
57-63. 

Abstract:  The  software  development  process  is 
looked  at  as  to  the  specific  contribution  of 
inspections/ reviews  to  the  discovery  of  wrong  de¬ 
sign  directions  or  implementations.  The  benefits  are 
evaluated  under  the  aspects  of  quality /productivity 
improvement  and/or  cost  savings. 

The  relationship  of  review  processes  to  testing  in  an 
IBM  environment  are  explored.  A  variation  of  the 
roles  of  the  review  participants  is  presented  as  well. 
The  utilization  of  defect  data  to  the  discovery  of 
wrong  design  directions  and  implementations  is  also 
described. 

Wa{ker79 

Walker,  M.  “Auditing  Software  Development  Proj¬ 
ects:  A  Control  Mechanism  for  the  Digital  Systems 


Development  Methodology.”  Procsedings,  COM- 
PCON  Spring.  Silver  Spring,  MD:  'lEEE  Computer 
Society  Press,  1979,  310-314. 

Software  audits  and  their  function  in  a  development 
organization  are  defined.  Auditing  techniques  arc 
presented  as  well  as  experiences  from  the  Computer 
Science  Corporation. 

Welnberg84 

Weinberg,  G.,  and  D.  Freedman.“Reviews,  Walk¬ 
throughs,  and  Inspections.”  IEEE  Trans.  Software 
Eng.  SE-IO,  1  (Jan.  1984),  68-72. 

Abstract:  Formal  technical  reviews  supply  the 
quality  measurement  to  the  "cost  effectiveness" 
equation  in  a  project  management  system.  There 
are  several  unique  formal  technical  review  proce¬ 
dures,  each  applicable  to  particular  types  of  tech¬ 
nical  material  and  to  the  particular  mix  of  the  Re¬ 
view  Committee.  All  formal  technical  reviews  pro¬ 
duce  reports  on  the  overall  quality  for  project  man¬ 
agement,  and  specific  technical  information  for  the 
producers.  These  reports  also  serve  as  an  historic 
account  of  the  systems  development  process.  His¬ 
toric  origins  and  future  trends  of  formal  and  infor¬ 
mal  technical  reviews  are  discus.sed. 

An  overview  paper  describing  the  distinction  be¬ 
tween  walkthroughs  and  inspections.  The  dif¬ 
ference  between  formal  and  infonnal  reviews  is  also 
clarified.  The  paper  also  contains  sample  review 
reports  and  how  these  repoits  can  Ibe  used. 

Yourdon78 

Yourdon,  E.  Structured  Walkthroughs.  New  York; 
Yourdon,  Inc.,  1978. 

A  detailed  discussion  of  the  walkthrough  process. 
The  benefits  of  walkthroughs  as  v/ell  as  the  me¬ 
chanics  of  the  process  are  presented.  Psychological 
issues  for  walkthroughs  are  also  note!. 


SEI-SM-3-1.0 


Draft  For  Public  Review 


75 


Contributors 


The  Software  Technical  Review  Process 


Appendix;:  Addresses  of  Contributors 


James  S.  Collofello 
Computer  Science  Deparanent 
Arizona  State  University 
Tcmpe,  AZ  85287 

CSnei:  coIlofe!@asu 

ARPAnei:  collofei%asa@relay.cs.net 

John  A.  Cross 

Department  of  Computer  Science 
Indiana  University  of  Pennsylvania 
Indiana,  PA  15705 

EITnet:  jacross(®iuo 

ARPAnet:  jacross%iup.bitnet@ vnuLCC.cmu.edu 


76 


Draft  For  Public  Ravlaw 


SEI-SM-3-1.0 


ttis  iia..i!saA5Jum» _ 

tfAif  V  iCAttdN  af  TMit  >*0I 


tMOMf  KCwMtt  V  <Ukl|«»tC«1iON 

wdhmfm 


•  IfwOt*  *  CV*U)f  i^AtlQN  AUlHOniTV  . 

N/A  ___ 


0fCW*ll'*'<>*^*O'^’00ViNOM«0lNQ  ICMlOUt.1 

N/A _ 


•  OftQAHI|*riON  HlPOMT  NWMaiflll) 

SRl-SN-i-I.O 


^~N*WI  «»  »(A»OIIMiNQ  pNOf^NllATION 

80FTWAAR  RNCINEERINC  INST. 


REPORT  OOCUMENTAUON  PAGE 


1«,  AltTKtCTtVC  MAAKINQI 


,  Aponita  iCfi*.  t««H  M*  xfA  cMti 

CARNRGIE  HRIUON  UNlVEIi:^lTY 
PITTSIURGH.  PA  15213 


MAMt  Of  »UNOiNQ/«*ON«OMlNQ 
OMQANllATlON 

SRI  JOINT  PKOGRAM  OFFICE 


AOeMIII  ««•«*  vs4  G»4tl 

CARNRGIE  HELION  UNIVERSIH 
PITTSBURGH,  PA  13213 


9  Oi|9AiauTlON/AVAIt>|lWlTV  0*  AlfiQAT 

APPROVED  FOR  PUBLIC  RELEASE 
DISTRIBUTION  UNLIMITED 


t.  M0»  irO'^INQ  OAQANitAflON  ACPOMT  NUM*tA(S) 


U  NAMt  Of  MONITOAINO  OAQANlXAnON 

SEX  JOINT  PROGRAM  OFFICE 


n,  AOOAtu  ie»t,  siti*  aa*  x/a  c«f«> 

ESD/AVS 

KANSCOM  AIR  FORCE  BASE.  MA  01731 


to.  Of  f  ICI  tVMtOU  •.  AAOCUAIMINT  INSTAUMINT  lOSNTIf  ICATION  NWMtIM 
Of 

ESD/AVS  n962890C0003 


10.  tOUACI  Of  f  UNOINO  NOS. 


IMONAW  AUTHOI^d) 


•  FI  iTiWiT®  Hnnw  fiT*  J7*cm  En.f  F 


19^  riMl  eoviAio 
f  AOM _  TO 


fAOOAAM 

fAOjicr 

TASK 

MOAK  UNIT 

CLIMINT  NO. 

NO. 

NO 

NO 

63752F 

M/A 

N/A 

N/A 

COSATI  COOIS 


14  SUtJiCT  TIAMS  <C«AnAu»  vn  nw«n*  tf  ftttutn  '"f  <4tnHfy  fy  S.o<f  nymtoN 

technical  review  walkthrough 


technical  review  walkthrough 

Inspection 


AMTA  act  fCfn<iny«  i,n  if  A«<tM4n’  tAf  idtntlff  Sy  Siott  Aymf«.; 

These  materials  support  the  SEI  curri.ulum  module  SEI-CM-3  "The  Software  Technical 
Review  Process." 


ll|TAieuTlON/AVAILASIi.lTV  Of  ASSTAaCT 

2t.  ASSTAACT  SICUAITV  CLASStf  ICATION 

i  (.ASSif  l■0/UN...MllTtO  ^  SAMI  AS  AfT.  □  OTIC  LSE^.«  CB 

UNCUSSIFIED,  UNLIMITED  DISTRIBUTION 

NAMI  Of  AtIfQNSllLl  INOIVIOUAU 

JOHN  S.  HERMAN,  Cape.  USAF 


’ORM  1473,  83  APR  soition  of  i  jan 


92A  TCLIfHONI  NUMSIA 
IfntM*  Alt*  C»4»f 


412  268-7630 


iOlTION  Of  1  JAN  ?9  IS  OeSOLfTI. 


22c.  Of  fICt  SVMtOk 

ESD/AVS 
JPO 


Um, IMITFI),  IWri.ASSTFTFD 


Th«  Softwar*  Engin««ring  Instiluta  (SEl)  is  a  ladafalty  lundad  rasaarch  and  developmant  canter,  oporatad  by  Carnagia 
MaHon  Univarsity  undar  contract  wKh  tha  UnHad  Stalas  Dapartmant  o(  Dalansa. 

Tha  SEl  Software  Enginaaring  Curriculum  Project  is  developing  a  wide  range  of  materials  to  support  software  engineering 
education.  A  currhulvm  modula  (CM)  idontifies  and  outlines  the  content  of  a  specific  topic  area,  and  is  intended  to  be 
used  by  an  instmetor  in  designing  a  course.  A  support  materials  package  (SM)  contains  materials  related  to  a  module 
that  may  be  helpful  in  teachir>g  a  course.  An  educational  materials  package  (EM)  contains  other  materials  r^l  necessarily 
ratated  to  a  curriculum  module.  Other  publications  include  software  engineering  curriculum  recommendations  and  course 
destgrrs. 

SEl  educational  materials  are  being  made  available  to  educators  throughout  the  academic,  industrial,  and  government 
communities.  The  use  of  these  materials  in  a  course  does  not  in  any  way  constitute  an  endorsement  of  the  course  by  the 
SEl,  by  Carnegie  Melon  University,  or  by  the  United  Slates  government 

Permission  to  make  copies  or  derivative  works  of  SEl  curriculum  modules,  support  materials,  and  educational  materials  is 
granted,  without  fee,  provided  that  the  copies  and  derivative  works  are  not  made  or  distributed  for  direct  commerciai 
advantage,  and  that  al  copies  and  derivative  works  cHe  the  original  documant  by  nama,  author's  name,  and  document 
number  and  give  notice  that  the  copying  Is  by  permission  of  Carnegie  Mellon  University. 

Comments  on  SEl  educational  materials  artd  requests  for  additional  information  should  be  addressed  to  the  Software 
Engineering  Curriculum  Project,  Software  Engineering  Institute.  Carnegie  Mellon  University,  Pittsburgh,  Pertrisyh/ania 
15213.  E'.«ctror.ic  mal  can  tM  sent  to  educatkx^sei.cmu.edu  on  the  Internet. 


Cuntailur/  Modules  (*  Support  MaMriais  available) 

CM-1  (superseded  by  CM-191 

CM-2  Intoducion  to  Software  Design 

CM4  The  Software  Teohnieal  fleview  Preotss* 

CM4  Software  ConRguraiion  Management* 

CM-S  Information  Piotociion 
CM4  Software  Safety 
CM>7  Assurance  of  SofAware  Oueliy 
CM-I  Formal  Spedicatfon  of  Software* 

CM4  Unk  Testing  and  Antoysia 

CM-10  Models  of  Software  Evolution:  Ufa  Cyde  and  Proceas 

CM-1 1  Software  Spedlicadons:  A  Framework 

CM-12  Software  Metrics 

CM- 1 3  Inboductfon  to  Software  Verilicatfon  and  Vafidabon 
CM-14  Intolectoaf  Property  Protoction  for  Software 
CM-1S  Software  Dovefopnreni  and  Lioensing  Contracts 
CM-16  Software  Davafopment  Using  VOM 
CM-17  User  fnlatface  Oevelopmanr 
CM- 18  [superseded  by  CM-23] 

CM-ig  Sofhvare  RequiramanU 

CM-20  Formal  Varificatiofl  of  Programs 

CM-21  Software  Project  Management 

CM-22  Soflware  Design  Methods  hr  Real-Time  Systems* 

CM-23  Technical  Writing  tor  Software  Engineers 

CM-24  Concepts  of  Cortcurrent  Programming 

CM-25  Language  and  System  Support  for  Concurrent 
Programming* 

CM  26  Undsrstandf^g  Program  Dependencies 


Educational  Materials 

EM-1  Softerara  Maintonanoe  Exerdses  for  a  Software 
Engineering  Project  Course 

EM-2  APSE  Intoracdve  Monitor  An  Artifact  (or  Software 
Enginaaring  Education 

EM-3  Reading  Computer  Programs:  Instructor's  Guide  and 
Exardsee 


